2024-03-29T10:33:43.488 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:33:43.488 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/orig.config.yaml 2024-03-29T10:33:43.521 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:33:43.617 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622651' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPN+qhibP5MMVarEo4s+gy8Sm4/97TldBKn8eK5G+fp0IKtrXNwB1gabHb7cTMmUTPY2wNZf3cU4EgfTWGG5M34= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLn8du5iTM/9h2XN3fmByzGpl7IBQMOUJR3iOwvLiEQM1xEK7UgdfKPxV2SN95N9JoBj4dZl5JKZauekhccJMRE= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:33:43.617 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:33:43.618 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:33:43.618 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:33:43.620 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:33:43.643 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:33:43.643 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:33:43.644 INFO:teuthology.packaging:ref: None 2024-03-29T10:33:43.644 INFO:teuthology.packaging:tag: None 2024-03-29T10:33:43.644 INFO:teuthology.packaging:branch: main 2024-03-29T10:33:43.644 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:33:43.645 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:33:44.007 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:33:44.008 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:33:44.016 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:33:44.016 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:33:44.035 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:33:44.049 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:33:44.057 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:33:44.083 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:28:35.707724', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPN+qhibP5MMVarEo4s+gy8Sm4/97TldBKn8eK5G+fp0IKtrXNwB1gabHb7cTMmUTPY2wNZf3cU4EgfTWGG5M34='} 2024-03-29T10:33:44.108 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:28:35.706312', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLn8du5iTM/9h2XN3fmByzGpl7IBQMOUJR3iOwvLiEQM1xEK7UgdfKPxV2SN95N9JoBj4dZl5JKZauekhccJMRE='} 2024-03-29T10:33:44.108 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:33:44.117 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:33:44.117 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:33:44.117 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:33:44.190 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc481eb90d0>, signals=[15]) 2024-03-29T10:33:44.190 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:33:44.200 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:33:44.200 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:33:44.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:44.289 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:33:44.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:44.377 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:33:44.385 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-03-29T10:33:44.402 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-03-29T10:33:44.402 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-03-29T10:33:44.461 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:33:44.462 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:33:44.463 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:33:44.463 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:33:44.463 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:33:44.463 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:33:44.463 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-03-29T10:33:44.485 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-03-29T10:33:44.502 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-03-29T10:33:44.503 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-03-29T10:33:44.558 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:33:44.559 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:33:44.560 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-03-29T10:33:44.585 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:33:44.594 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:33:44.606 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:33:44.606 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:33:44.609 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:33:44.625 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:33:44.631 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:33:44.631 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:33:44.669 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:33:44.761 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:33:44.819 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:33:44.834 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:33:44.834 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:33:44.834 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:33:44.835 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:33:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:33:44.850 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-03-29T10:33:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:33:44.851 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-03-29T10:33:44.911 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:33:44.911 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-361.el9.x86_64 2024-03-29T10:33:44.911 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-03-29T10:33:44.912 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:33:44.912 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.14.0-361.el9.x86_64 2024-03-29T10:33:44.913 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum install -y kernel 2024-03-29T10:33:47.047 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:33:47.177 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T10:33:48.603 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-29T10:33:48.881 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-29T10:33:50.151 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-03-29T10:33:52.680 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-29T10:33:53.224 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-03-29T10:33:56.736 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T10:33:59.186 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T10:34:00.863 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T10:34:02.112 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-29T10:34:05.787 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T10:34:06.298 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-29T10:34:08.094 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:34:08.094 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:34:08.183 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:34:08.192 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:34:08.192 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:34:08.192 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:34:08.193 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:34:08.193 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:34:08.193 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:34:08.193 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-03-29T10:34:08.194 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:08.195 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 76 M 2024-03-29T10:34:08.195 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 123 M 2024-03-29T10:34:08.195 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:34:08.338 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:34:08.671 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-29T10:34:08.838 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T10:34:09.355 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 29 MB/s | 34 MB 00:01 2024-03-29T10:34:09.714 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:34:09.717 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:09.717 INFO:teuthology.orchestra.run.smithi079.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-29T10:34:09.717 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:34:10.093 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:34:10.093 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:34:10.923 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:34:10.923 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:34:13.237 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:34:14.044 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:34:14.139 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:34:16.896 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:34:17.031 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:34:21.734 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:34:21.804 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:34:26.520 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:34:55.959 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:34:55.959 INFO:teuthology.orchestra.run.smithi079.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:34:55.959 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:56.003 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:34:56.423 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:34:56.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:34:56.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:34:56.424 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:34:56.837 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:34:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:56.838 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:34:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:34:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:34:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:34:56.838 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:34:56.839 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:56.839 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:34:57.163 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:34:57.259 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:34:57.259 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:34:57.260 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:34:57.260 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:34:57.260 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:34:57.260 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:34:57.260 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:34:57.260 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:34:57.339 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:34:57.340 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:34:57.340 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-03-29T10:34:57.374 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 68.el9 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8264254 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-03-29T10:34:57.375 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:34:57.376 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:34:57.377 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:34:57.377 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:34:57.377 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:34:57.997 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-03-29T10:34:59.404 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:34:59.448 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-03-29T10:34:59.451 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-03-29T10:34:59.466 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.Pa2fWljqfY 2024-03-29T10:34:59.466 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Pa2fWljqfY 2024-03-29T10:34:59.535 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.Pa2fWljqfY 2024-03-29T10:34:59.675 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.Pa2fWljqfY is 6KB 2024-03-29T10:34:59.723 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.Pa2fWljqfY 2024-03-29T10:34:59.737 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:34:59.802 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:34:59.802 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:34:59.802 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:34:59.802 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:34:59.803 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:34:59.942 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-03-29T10:35:29.945 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:35:29.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:35:29.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:48.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-29T10:35:57.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:35:57.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:36:00.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-29T10:36:12.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:36:12.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:36:30.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-03-29T10:36:35.974 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 144 kB/s | 21 MB 02:27 2024-03-29T10:36:40.865 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T10:36:43.130 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:36:43.130 INFO:teuthology.orchestra.run.smithi082.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:36:43.222 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:36:43.230 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout:Install 4 Packages 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 76 M 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 123 M 2024-03-29T10:36:43.231 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:36:43.374 INFO:teuthology.orchestra.run.smithi082.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:36:43.958 INFO:teuthology.orchestra.run.smithi082.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T10:36:44.592 INFO:teuthology.orchestra.run.smithi082.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 23 MB/s | 28 MB 00:01 2024-03-29T10:36:44.908 INFO:teuthology.orchestra.run.smithi082.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T10:36:44.912 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:44.912 INFO:teuthology.orchestra.run.smithi082.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T10:36:44.912 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:36:45.277 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:36:45.277 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:36:45.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:36:45.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:36:46.103 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:36:46.103 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:36:46.367 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-29T10:36:46.867 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:36:46.867 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:36:46.868 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-03-29T10:36:46.881 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:36:46.882 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:36:46.882 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:36:46.882 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:36:47.883 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:36:47.883 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:36:47.968 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-03-29T10:36:47.995 DEBUG:teuthology.parallel:result is None 2024-03-29T10:36:48.431 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:36:49.087 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:36:49.175 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:51.975 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:36:52.112 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:56.841 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:36:56.935 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:01.674 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:31.356 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:31.356 INFO:teuthology.orchestra.run.smithi082.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:37:31.356 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:37:31.398 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:31.744 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:31.744 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:37:31.744 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:37:31.744 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:37:32.125 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:37:32.126 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:37:32.445 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:37:32.548 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:37:32.548 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:37:32.548 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:37:32.548 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:37:32.549 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:37:32.549 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:37:32.549 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:37:32.549 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:37:32.628 INFO:teuthology.orchestra.run.smithi082.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:37:32.628 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:37:32.628 DEBUG:teuthology.orchestra.run.smithi082:> sudo rpm -qi grub2-tools 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Name : grub2-tools 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Epoch : 1 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Version : 2.06 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Release : 68.el9 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: x86_64 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Group : Unspecified 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Size : 8264254 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:License : GPLv3+ 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:37:32.665 INFO:teuthology.orchestra.run.smithi082.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:Packager : builder@centos.org 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:Vendor : CentOS 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:Summary : Support tools for GRUB. 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:Description : 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:hardware devices. 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:37:32.666 INFO:teuthology.orchestra.run.smithi082.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:37:32.667 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:37:32.667 INFO:teuthology.task.kernel:Updating grub on smithi082 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:37:32.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:37:33.240 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-03-29T10:37:34.622 INFO:teuthology.orchestra.run.smithi082.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:37:34.662 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-03-29T10:37:34.664 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-03-29T10:37:34.679 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.xnXhGsqfPK 2024-03-29T10:37:34.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xnXhGsqfPK 2024-03-29T10:37:34.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.xnXhGsqfPK 2024-03-29T10:37:34.886 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.xnXhGsqfPK is 6KB 2024-03-29T10:37:34.934 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.xnXhGsqfPK 2024-03-29T10:37:34.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:37:35.012 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:37:35.013 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:37:35.013 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:37:35.013 INFO:teuthology.orchestra.run.smithi082.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:37:35.014 DEBUG:teuthology.orchestra.run.smithi082:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:37:35.164 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-03-29T10:38:05.167 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:38:05.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T10:38:05.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:38:39.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-29T10:38:48.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T10:38:48.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:39:07.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-29T10:39:19.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T10:39:19.044 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:39:19.384 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-29T10:39:19.772 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T10:39:19.772 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:39:19.772 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-03-29T10:39:19.786 INFO:teuthology.orchestra.run.smithi082.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:39:19.786 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:39:19.786 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:39:19.786 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:39:20.787 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:39:20.788 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:39:20.867 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-03-29T10:39:20.907 DEBUG:teuthology.parallel:result is None 2024-03-29T10:39:20.907 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:39:20.914 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:39:20.915 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:39:20.918 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:39:20.934 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:39:20.942 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:39:20.954 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:39:20.954 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:39:20.976 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:39:21.065 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:39:21.073 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:39:21.074 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:39:21.076 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:39:21.115 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:39:21.130 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:39:21.148 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:39:21.161 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:39:21.163 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:39:21.172 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:39:21.172 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:39:21.175 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:39:21.203 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:39:21.213 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:39:21.213 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:39:21.246 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:39:21.260 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:39:21.342 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:39:21.392 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:39:21.469 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:39:21.518 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:39:21.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:39:21.586 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:39:21.629 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:39:21.675 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:39:21.754 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:39:21.801 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:39:21.801 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:39:21.870 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-03-29T10:39:21.872 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-03-29T10:39:21.929 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:39:21.984 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:39:22.479 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:39:22.487 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:39:22.487 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:39:22.499 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:39:22.511 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-03-29T10:39:22.565 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-03-29T10:39:22.566 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-03-29T10:39:22.670 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-03-29T10:39:22.671 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:39:22.672 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-03-29T10:39:22.692 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-03-29T10:39:22.692 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-03-29T10:39:22.719 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-03-29T10:39:22.719 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:39:22.719 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:39:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:39:22.760 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:39:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:39:22.788 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:39:22.788 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:39:22.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:39:22.871 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:39:22.880 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:39:22.897 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:39:22.899 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:44:35.157 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-03-29T10:44:35.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:44:35.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:35.247 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-03-29T10:44:35.329 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-03-29T10:44:35.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T10:44:35.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:35.419 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-29T10:44:35.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-29T10:44:35.502 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:44:35.512 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:44:35.513 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:44:35.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:44:35.515 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:44:35.515 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:44:35.544 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:44:35.560 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:44:35.574 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:44:35.589 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-03-29T10:44:35.591 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:44:35.603 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-03-29T10:44:35.635 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-03-29T10:44:35.636 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:44:35.652 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:44:35.668 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:44:35.689 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:44:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:44:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-29T10:44:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.693 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.718 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:44:35.722 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:44:35.722 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-29T10:44:35.722 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.722 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.722 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.723 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:44:35.723 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:44:35.731 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:44:35.731 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:44:35.731 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:44:35.731 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:44:35.737 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:44:35.737 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:44:35.737 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:44:35.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:44:35.739 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:44:35.740 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:44:35.883 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:44:35.883 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:44:35.954 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:44:35.954 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:44:36.052 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:44:36.053 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:44:36.053 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:44:36.080 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:44:36.081 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:44:36.121 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:44:36.121 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:44:36.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:44:36.149 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:44:36.150 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:44:36.150 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:44:36.219 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:44:36.229 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:44:36.296 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-03-29T10:44:36.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-03-29T10:44:36.299 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:44:36.366 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-03-29T10:44:36.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-03-29T10:44:36.541 INFO:teuthology.orchestra.run.smithi082.stdout:74 files removed 2024-03-29T10:44:36.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw 2024-03-29T10:44:36.612 INFO:teuthology.orchestra.run.smithi079.stdout:74 files removed 2024-03-29T10:44:36.639 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-03-29T10:44:37.240 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 219 kB/s | 82 kB 00:00 2024-03-29T10:44:37.401 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 177 kB/s | 82 kB 00:00 2024-03-29T10:44:37.665 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T10:44:37.751 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-29T10:44:38.016 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-29T10:44:38.210 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-03-29T10:44:38.477 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:44:38.654 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T10:44:40.052 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-29T10:44:40.263 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-29T10:44:41.309 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-03-29T10:44:41.362 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:45:03.681 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 909 kB/s | 19 MB 00:21 2024-03-29T10:45:03.917 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 900 kB/s | 19 MB 00:21 2024-03-29T10:45:07.531 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-29T10:45:07.984 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T10:45:09.166 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T10:45:09.576 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-29T10:45:12.425 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-03-29T10:45:12.486 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-29T10:45:17.283 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-29T10:45:17.452 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T10:45:19.896 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:45:19.911 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:45:19.911 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:45:19.911 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:45:19.911 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:45:19.911 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:45:19.912 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:45:19.913 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:19.914 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:45:19.915 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:45:19.915 INFO:teuthology.orchestra.run.smithi082.stdout:Install 30 Packages 2024-03-29T10:45:19.915 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 2 Packages 2024-03-29T10:45:19.915 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:19.915 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 58 M 2024-03-29T10:45:19.915 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:45:20.054 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:45:20.075 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:45:20.075 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:45:20.076 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:45:20.077 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-03-29T10:45:20.078 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:Install 30 Packages 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 58 M 2024-03-29T10:45:20.079 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:45:20.500 INFO:teuthology.orchestra.run.smithi082.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 10 MB/s | 5.3 MB 00:00 2024-03-29T10:45:20.542 INFO:teuthology.orchestra.run.smithi082.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 605 kB/s | 25 kB 00:00 2024-03-29T10:45:20.617 INFO:teuthology.orchestra.run.smithi082.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 9.7 MB/s | 742 kB 00:00 2024-03-29T10:45:20.676 INFO:teuthology.orchestra.run.smithi082.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 8.3 MB/s | 498 kB 00:00 2024-03-29T10:45:21.118 INFO:teuthology.orchestra.run.smithi082.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 12 MB/s | 5.1 MB 00:00 2024-03-29T10:45:21.176 INFO:teuthology.orchestra.run.smithi082.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 774 kB/s | 45 kB 00:00 2024-03-29T10:45:21.243 INFO:teuthology.orchestra.run.smithi082.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T10:45:21.310 INFO:teuthology.orchestra.run.smithi082.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-29T10:45:21.394 INFO:teuthology.orchestra.run.smithi082.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T10:45:21.477 INFO:teuthology.orchestra.run.smithi082.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T10:45:21.561 INFO:teuthology.orchestra.run.smithi082.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T10:45:21.669 INFO:teuthology.orchestra.run.smithi082.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-29T10:45:21.711 INFO:teuthology.orchestra.run.smithi082.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T10:45:21.745 INFO:teuthology.orchestra.run.smithi082.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T10:45:21.796 INFO:teuthology.orchestra.run.smithi082.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T10:45:21.828 INFO:teuthology.orchestra.run.smithi079.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.2 MB/s | 9.7 MB 00:01 2024-03-29T10:45:21.829 INFO:teuthology.orchestra.run.smithi082.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-29T10:45:21.862 INFO:teuthology.orchestra.run.smithi082.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T10:45:21.870 INFO:teuthology.orchestra.run.smithi079.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 607 kB/s | 25 kB 00:00 2024-03-29T10:45:21.904 INFO:teuthology.orchestra.run.smithi082.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-03-29T10:45:21.930 INFO:teuthology.orchestra.run.smithi082.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T10:45:21.963 INFO:teuthology.orchestra.run.smithi082.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T10:45:21.987 INFO:teuthology.orchestra.run.smithi079.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T10:45:22.054 INFO:teuthology.orchestra.run.smithi079.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T10:45:22.414 INFO:teuthology.orchestra.run.smithi082.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 686 kB/s | 309 kB 00:00 2024-03-29T10:45:22.564 INFO:teuthology.orchestra.run.smithi082.stdout:(22/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.7 MB/s | 9.7 MB 00:02 2024-03-29T10:45:22.571 INFO:teuthology.orchestra.run.smithi079.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.9 MB/s | 5.1 MB 00:00 2024-03-29T10:45:22.622 INFO:teuthology.orchestra.run.smithi079.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 906 kB/s | 45 kB 00:00 2024-03-29T10:45:22.719 INFO:teuthology.orchestra.run.smithi079.stdout:(7/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.4 MB/s | 5.3 MB 00:02 2024-03-29T10:45:22.747 INFO:teuthology.orchestra.run.smithi079.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.0 MB/s | 131 kB 00:00 2024-03-29T10:45:22.780 INFO:teuthology.orchestra.run.smithi079.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.5 MB/s | 162 kB 00:00 2024-03-29T10:45:22.790 INFO:teuthology.orchestra.run.smithi082.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-29T10:45:22.808 INFO:teuthology.orchestra.run.smithi079.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.3 MB/s | 320 kB 00:00 2024-03-29T10:45:22.847 INFO:teuthology.orchestra.run.smithi079.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.3 MB/s | 298 kB 00:00 2024-03-29T10:45:22.856 INFO:teuthology.orchestra.run.smithi082.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T10:45:22.873 INFO:teuthology.orchestra.run.smithi079.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 99 kB 00:00 2024-03-29T10:45:22.948 INFO:teuthology.orchestra.run.smithi082.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 736 kB/s | 67 kB 00:00 2024-03-29T10:45:22.965 INFO:teuthology.orchestra.run.smithi082.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-03-29T10:45:23.015 INFO:teuthology.orchestra.run.smithi079.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 106 kB 00:00 2024-03-29T10:45:23.040 INFO:teuthology.orchestra.run.smithi079.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 954 kB/s | 159 kB 00:00 2024-03-29T10:45:23.057 INFO:teuthology.orchestra.run.smithi079.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T10:45:23.065 INFO:teuthology.orchestra.run.smithi082.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-29T10:45:23.116 INFO:teuthology.orchestra.run.smithi079.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T10:45:23.141 INFO:teuthology.orchestra.run.smithi079.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T10:45:23.157 INFO:teuthology.orchestra.run.smithi082.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 998 kB/s | 192 kB 00:00 2024-03-29T10:45:23.158 INFO:teuthology.orchestra.run.smithi079.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2024-03-29T10:45:23.183 INFO:teuthology.orchestra.run.smithi079.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:45:23.200 INFO:teuthology.orchestra.run.smithi079.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 837 kB/s | 35 kB 00:00 2024-03-29T10:45:23.217 INFO:teuthology.orchestra.run.smithi079.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T10:45:23.375 INFO:teuthology.orchestra.run.smithi079.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-03-29T10:45:23.409 INFO:teuthology.orchestra.run.smithi079.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 780 kB/s | 26 kB 00:00 2024-03-29T10:45:23.443 INFO:teuthology.orchestra.run.smithi079.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-29T10:45:23.485 INFO:teuthology.orchestra.run.smithi079.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T10:45:23.693 INFO:teuthology.orchestra.run.smithi079.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-29T10:45:23.699 INFO:teuthology.orchestra.run.smithi082.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 5.3 MB/s | 20 MB 00:03 2024-03-29T10:45:23.766 INFO:teuthology.orchestra.run.smithi082.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.5 MB/s | 3.4 MB 00:00 2024-03-29T10:45:23.768 INFO:teuthology.orchestra.run.smithi079.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T10:45:23.808 INFO:teuthology.orchestra.run.smithi082.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-29T10:45:23.966 INFO:teuthology.orchestra.run.smithi082.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-03-29T10:45:23.969 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:23.969 INFO:teuthology.orchestra.run.smithi082.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T10:45:23.970 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:45:24.003 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:45:24.003 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:45:24.069 INFO:teuthology.orchestra.run.smithi079.stdout:(28/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 5.5 MB/s | 20 MB 00:03 2024-03-29T10:45:24.353 INFO:teuthology.orchestra.run.smithi079.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-03-29T10:45:24.428 INFO:teuthology.orchestra.run.smithi079.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-03-29T10:45:24.470 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:45:24.471 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:45:24.653 INFO:teuthology.orchestra.run.smithi079.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-03-29T10:45:24.803 INFO:teuthology.orchestra.run.smithi079.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.6 MB/s | 3.4 MB 00:00 2024-03-29T10:45:24.807 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:24.807 INFO:teuthology.orchestra.run.smithi079.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-29T10:45:24.808 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:45:24.841 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:45:24.841 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:45:25.326 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:45:25.326 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:45:25.919 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:45:26.006 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:45:26.037 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:45:26.062 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:45:26.094 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:45:26.109 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:45:26.311 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:45:26.394 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:45:26.741 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:45:26.789 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:45:26.811 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:45:26.863 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:45:26.897 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:45:26.933 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:45:26.951 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:45:26.958 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:45:26.974 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:45:26.989 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:45:27.006 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:45:27.036 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:45:27.057 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:45:27.092 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:45:27.159 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:45:27.184 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:45:27.189 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:45:27.244 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:45:27.247 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:45:27.282 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:45:27.390 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:45:27.442 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:45:27.483 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:45:27.516 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:45:27.545 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:45:27.575 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:45:27.577 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:45:27.626 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:45:27.721 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:45:27.809 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:45:27.845 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:45:27.891 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:45:27.910 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:45:27.920 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:45:27.932 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:45:27.964 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:45:28.022 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:45:28.031 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:45:28.053 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:45:28.055 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:45:28.102 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:45:28.105 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:45:28.138 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:45:28.245 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:45:28.280 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:45:28.321 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:45:28.353 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:45:28.382 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:45:28.415 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:45:28.579 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:45:28.609 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:45:28.680 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:45:28.737 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:45:28.748 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:45:28.762 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:45:28.778 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:45:28.834 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:45:28.867 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:45:28.910 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:45:29.365 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:45:29.389 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:45:29.444 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:45:29.511 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:45:29.542 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:45:30.882 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:45:30.924 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:45:31.474 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:45:31.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:45:31.594 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:45:31.594 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:45:31.594 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:45:31.594 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:31.637 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:45:31.761 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:45:31.796 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:45:32.343 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:45:32.383 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:45:32.477 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:45:32.477 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:45:32.477 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:45:32.478 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:32.526 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-03-29T10:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-03-29T10:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-03-29T10:45:41.109 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-03-29T10:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-03-29T10:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-03-29T10:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-03-29T10:45:41.110 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:41.993 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:45:42.018 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:45:42.018 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:42.018 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:45:42.018 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:45:42.018 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:45:42.018 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:42.019 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:45:42.120 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:45:42.120 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:45:42.140 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:45:42.140 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-03-29T10:45:42.140 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-03-29T10:45:42.140 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-03-29T10:45:42.141 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-03-29T10:45:42.141 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-03-29T10:45:42.141 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-03-29T10:45:42.141 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-03-29T10:45:42.141 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:43.057 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:45:43.081 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:45:43.082 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:45:43.082 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:45:43.082 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:45:43.082 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:45:43.082 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:43.084 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:45:43.160 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:45:43.160 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:45:43.254 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:45:43.255 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:45:43.256 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:45:44.232 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:45:44.233 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:44.235 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:45:44.333 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:45:44.333 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:45:44.333 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:45:44.333 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:45:44.333 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:45:44.333 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:45:44.334 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:45:44.335 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:45:44.336 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:45:44.336 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:45:44.336 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:45:44.336 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:45:44.595 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-test 2024-03-29T10:45:45.167 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:45:45.317 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:45:45.317 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:45:45.318 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:45.319 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:45:45.319 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:45:45.319 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-03-29T10:45:45.319 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:45.319 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 48 M 2024-03-29T10:45:45.320 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 204 M 2024-03-29T10:45:45.320 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:45:45.396 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.397 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:45:45.398 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:45.399 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:45:45.518 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:45:45.569 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:45:45.602 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:45:45.636 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 581 kB/s | 163 kB 00:00 2024-03-29T10:45:45.752 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-03-29T10:45:45.769 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-29T10:45:46.306 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:45:46.465 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:45:46.466 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-03-29T10:45:46.467 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:46.468 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 48 M 2024-03-29T10:45:46.468 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 204 M 2024-03-29T10:45:46.468 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:45:46.742 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:45:46.784 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T10:45:46.826 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T10:45:47.227 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 253 kB/s | 163 kB 00:00 2024-03-29T10:45:47.685 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 218 kB/s | 187 kB 00:00 2024-03-29T10:45:47.761 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T10:45:47.764 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:47.765 INFO:teuthology.orchestra.run.smithi082.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-29T10:45:47.779 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:45:47.800 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:45:47.800 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:45:48.187 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:45:48.187 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:45:48.768 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:45:48.853 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:45:48.894 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:45:49.000 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:45:49.045 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:45:49.073 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:45:49.968 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 14 MB/s | 47 MB 00:03 2024-03-29T10:45:49.971 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:45:49.972 INFO:teuthology.orchestra.run.smithi079.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-29T10:45:49.986 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:45:50.008 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:45:50.008 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:45:50.389 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:45:50.390 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:45:50.958 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:45:51.036 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:45:51.086 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:45:51.169 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:45:51.211 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:45:51.239 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:45:53.553 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:45:53.631 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:45:55.701 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:45:55.789 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:45:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:45:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:45:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:45:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:45:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:45:56.027 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:45:56.370 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:56.371 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:45:56.624 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph 2024-03-29T10:45:57.208 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:45:57.371 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:45:57.373 INFO:teuthology.orchestra.run.smithi082.stdout:========================================================================================= 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout:========================================================================================= 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:45:57.374 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:45:57.375 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:45:57.376 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout:========================================================================================= 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout:Install 36 Packages 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:45:57.377 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 30 M 2024-03-29T10:45:57.378 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 107 M 2024-03-29T10:45:57.378 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:45:58.031 INFO:teuthology.orchestra.run.smithi082.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T10:45:58.173 INFO:teuthology.orchestra.run.smithi082.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-03-29T10:45:58.184 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:45:58.184 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:45:58.184 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:45:58.184 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:45:58.184 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:45:58.184 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:45:58.206 INFO:teuthology.orchestra.run.smithi082.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-03-29T10:45:58.298 INFO:teuthology.orchestra.run.smithi082.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.6 MB/s | 247 kB 00:00 2024-03-29T10:45:58.416 INFO:teuthology.orchestra.run.smithi082.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T10:45:58.457 INFO:teuthology.orchestra.run.smithi082.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:45:58.491 INFO:teuthology.orchestra.run.smithi082.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-29T10:45:58.524 INFO:teuthology.orchestra.run.smithi082.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:45:58.558 INFO:teuthology.orchestra.run.smithi082.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:45:58.588 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:58.589 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:45:58.600 INFO:teuthology.orchestra.run.smithi082.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:45:58.675 INFO:teuthology.orchestra.run.smithi082.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:45:58.709 INFO:teuthology.orchestra.run.smithi082.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:45:58.743 INFO:teuthology.orchestra.run.smithi082.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:45:58.776 INFO:teuthology.orchestra.run.smithi082.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T10:45:58.926 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-03-29T10:45:58.960 INFO:teuthology.orchestra.run.smithi082.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 21 MB/s | 17 MB 00:00 2024-03-29T10:45:59.160 INFO:teuthology.orchestra.run.smithi082.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 325 kB/s | 125 kB 00:00 2024-03-29T10:45:59.506 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:45:59.594 INFO:teuthology.orchestra.run.smithi082.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-03-29T10:45:59.675 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:45:59.677 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-03-29T10:45:59.677 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:45:59.678 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:45:59.679 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:45:59.680 INFO:teuthology.orchestra.run.smithi079.stdout:========================================================================================= 2024-03-29T10:45:59.681 INFO:teuthology.orchestra.run.smithi079.stdout:Install 36 Packages 2024-03-29T10:45:59.681 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:45:59.681 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 30 M 2024-03-29T10:45:59.681 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 107 M 2024-03-29T10:45:59.682 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:45:59.802 INFO:teuthology.orchestra.run.smithi082.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 235 kB/s | 151 kB 00:00 2024-03-29T10:45:59.878 INFO:teuthology.orchestra.run.smithi082.stdout:(19/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.5 MB/s | 4.7 MB 00:01 2024-03-29T10:46:00.003 INFO:teuthology.orchestra.run.smithi082.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 106 kB/s | 43 kB 00:00 2024-03-29T10:46:00.111 INFO:teuthology.orchestra.run.smithi082.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 557 kB/s | 172 kB 00:00 2024-03-29T10:46:00.283 INFO:teuthology.orchestra.run.smithi079.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-29T10:46:00.475 INFO:teuthology.orchestra.run.smithi079.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.1 MB/s | 1.5 MB 00:00 2024-03-29T10:46:00.595 INFO:teuthology.orchestra.run.smithi082.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 37 kB/s | 18 kB 00:00 2024-03-29T10:46:00.612 INFO:teuthology.orchestra.run.smithi082.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 18 kB/s | 11 kB 00:00 2024-03-29T10:46:00.687 INFO:teuthology.orchestra.run.smithi082.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-29T10:46:00.712 INFO:teuthology.orchestra.run.smithi082.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-29T10:46:00.780 INFO:teuthology.orchestra.run.smithi082.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-29T10:46:00.800 INFO:teuthology.orchestra.run.smithi079.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 9.1 MB/s | 4.7 MB 00:00 2024-03-29T10:46:00.876 INFO:teuthology.orchestra.run.smithi079.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-03-29T10:46:00.901 INFO:teuthology.orchestra.run.smithi079.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T10:46:00.938 INFO:teuthology.orchestra.run.smithi082.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 205 kB/s | 46 kB 00:00 2024-03-29T10:46:01.026 INFO:teuthology.orchestra.run.smithi079.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:46:01.038 INFO:teuthology.orchestra.run.smithi082.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 309 kB/s | 358 kB 00:01 2024-03-29T10:46:01.052 INFO:teuthology.orchestra.run.smithi079.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T10:46:01.064 INFO:teuthology.orchestra.run.smithi082.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 278 kB/s | 79 kB 00:00 2024-03-29T10:46:01.068 INFO:teuthology.orchestra.run.smithi079.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 751 kB/s | 32 kB 00:00 2024-03-29T10:46:01.085 INFO:teuthology.orchestra.run.smithi079.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:46:01.103 INFO:teuthology.orchestra.run.smithi079.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:46:01.128 INFO:teuthology.orchestra.run.smithi079.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:46:01.178 INFO:teuthology.orchestra.run.smithi079.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:46:01.203 INFO:teuthology.orchestra.run.smithi079.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:46:01.229 INFO:teuthology.orchestra.run.smithi079.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:46:01.262 INFO:teuthology.orchestra.run.smithi079.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T10:46:01.438 INFO:teuthology.orchestra.run.smithi079.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 17 MB/s | 17 MB 00:00 2024-03-29T10:46:01.472 INFO:teuthology.orchestra.run.smithi079.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 515 kB/s | 125 kB 00:00 2024-03-29T10:46:01.614 INFO:teuthology.orchestra.run.smithi079.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2024-03-29T10:46:01.647 INFO:teuthology.orchestra.run.smithi079.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 248 kB/s | 43 kB 00:00 2024-03-29T10:46:01.689 INFO:teuthology.orchestra.run.smithi079.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 604 kB/s | 151 kB 00:00 2024-03-29T10:46:01.748 INFO:teuthology.orchestra.run.smithi079.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 181 kB/s | 11 kB 00:00 2024-03-29T10:46:01.807 INFO:teuthology.orchestra.run.smithi079.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-29T10:46:01.832 INFO:teuthology.orchestra.run.smithi079.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-03-29T10:46:01.857 INFO:teuthology.orchestra.run.smithi079.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 706 kB/s | 172 kB 00:00 2024-03-29T10:46:01.883 INFO:teuthology.orchestra.run.smithi079.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T10:46:01.889 INFO:teuthology.orchestra.run.smithi082.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 19 kB/s | 16 kB 00:00 2024-03-29T10:46:01.899 INFO:teuthology.orchestra.run.smithi079.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-03-29T10:46:01.914 INFO:teuthology.orchestra.run.smithi082.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 106 kB/s | 90 kB 00:00 2024-03-29T10:46:01.916 INFO:teuthology.orchestra.run.smithi079.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 334 kB/s | 20 kB 00:00 2024-03-29T10:46:01.939 INFO:teuthology.orchestra.run.smithi082.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 272 kB/s | 272 kB 00:01 2024-03-29T10:46:01.942 INFO:teuthology.orchestra.run.smithi079.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 789 kB/s | 46 kB 00:00 2024-03-29T10:46:01.967 INFO:teuthology.orchestra.run.smithi079.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T10:46:01.992 INFO:teuthology.orchestra.run.smithi079.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-03-29T10:46:02.018 INFO:teuthology.orchestra.run.smithi079.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-29T10:46:02.043 INFO:teuthology.orchestra.run.smithi079.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T10:46:02.060 INFO:teuthology.orchestra.run.smithi079.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 532 kB/s | 36 kB 00:00 2024-03-29T10:46:02.065 INFO:teuthology.orchestra.run.smithi082.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 204 kB/s | 36 kB 00:00 2024-03-29T10:46:02.085 INFO:teuthology.orchestra.run.smithi079.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-29T10:46:02.110 INFO:teuthology.orchestra.run.smithi079.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-03-29T10:46:02.136 INFO:teuthology.orchestra.run.smithi079.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T10:46:02.140 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:02.140 INFO:teuthology.orchestra.run.smithi079.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-29T10:46:02.148 INFO:teuthology.orchestra.run.smithi082.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 982 kB/s | 230 kB 00:00 2024-03-29T10:46:02.164 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:02.206 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:02.206 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:02.207 INFO:teuthology.orchestra.run.smithi082.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-29T10:46:02.310 INFO:teuthology.orchestra.run.smithi082.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 81 kB/s | 20 kB 00:00 2024-03-29T10:46:02.313 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:02.314 INFO:teuthology.orchestra.run.smithi082.stdout:Total 6.0 MB/s | 30 MB 00:04 2024-03-29T10:46:02.338 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:02.379 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:02.379 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:02.608 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:02.609 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:02.794 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:02.794 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:03.939 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:04.005 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:46:04.040 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:46:04.074 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:46:04.113 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:04.145 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:46:04.168 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:46:04.182 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:46:04.209 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:46:04.214 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:46:04.237 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:46:04.305 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:46:04.307 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:46:04.353 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:46:04.377 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:46:04.402 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:46:04.418 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:46:04.489 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:46:04.493 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:46:04.544 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:46:04.572 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:46:04.582 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:46:04.613 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:46:04.631 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:46:04.684 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:46:04.687 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:46:04.739 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:46:04.777 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:46:04.826 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:46:04.872 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:46:04.880 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:46:04.921 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:46:04.968 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:46:05.002 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:46:05.032 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:46:05.064 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:46:05.101 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:46:05.114 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:46:05.135 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:46:05.144 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:46:05.168 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:46:05.179 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:46:05.210 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:46:05.225 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:46:05.261 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:46:05.285 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:46:05.295 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:46:05.320 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:46:05.353 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:46:05.384 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:46:05.394 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:46:05.426 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:46:05.428 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:46:05.463 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:46:05.499 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:46:05.544 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:46:05.586 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:46:05.636 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:46:05.657 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:46:05.771 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:46:05.793 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:46:05.920 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:46:05.935 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:46:05.966 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:46:05.991 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:46:05.991 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:05.991 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:46:05.991 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:46:05.991 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:46:05.991 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:06.077 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:46:06.108 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:46:06.133 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:46:06.133 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:06.133 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:46:06.134 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:46:06.134 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:46:06.134 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:07.416 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:46:07.442 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:46:07.443 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:07.443 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:46:07.443 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:46:07.443 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:46:07.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:07.567 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:46:07.594 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:46:07.594 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:07.594 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:46:07.594 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:46:07.594 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:46:07.594 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:07.884 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:46:07.906 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:46:07.906 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:07.907 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:46:07.907 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:46:07.907 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:46:07.907 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:08.043 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:46:08.066 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:46:08.066 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:08.066 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:46:08.066 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:46:08.066 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:46:08.067 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:08.109 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:46:08.132 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:46:08.132 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:08.132 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:46:08.133 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:46:08.133 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:46:08.133 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:08.198 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:46:08.302 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:46:08.325 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:46:08.325 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:08.325 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:46:08.325 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:46:08.325 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:46:08.325 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:08.391 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:46:09.939 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:46:09.939 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:46:09.940 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:46:09.941 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:46:09.942 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:46:09.942 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:46:09.942 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:46:09.943 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:46:09.943 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:46:09.943 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:46:09.943 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:46:09.943 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:46:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:46:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:46:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:46:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:46:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:46:10.146 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:46:10.147 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:46:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:46:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:46:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:46:10.148 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:46:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:46:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:46:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:46:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:46:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:46:10.149 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:10.864 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:46:10.865 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:46:10.866 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:46:10.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:46:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:46:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:46:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:46:10.868 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:10.868 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:11.122 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:46:11.123 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:46:11.124 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:46:11.125 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:46:11.126 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:46:11.126 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:11.126 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:11.214 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-03-29T10:46:11.451 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-base 2024-03-29T10:46:11.792 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:11.890 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:11.942 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:11.943 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:11.943 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:12.015 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:12.024 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-03-29T10:46:12.117 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:12.167 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:12.168 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:12.168 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:12.231 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install cephadm 2024-03-29T10:46:12.580 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:12.728 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:12.729 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:12.729 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:12.729 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:12.729 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:12.729 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 747 k 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 754 k 2024-03-29T10:46:12.730 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:12.787 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:12.936 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:12.937 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:46:12.938 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:12.938 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 747 k 2024-03-29T10:46:12.938 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 754 k 2024-03-29T10:46:12.938 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:13.004 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.7 MB/s | 747 kB 00:00 2024-03-29T10:46:13.005 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:13.005 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.7 MB/s | 747 kB 00:00 2024-03-29T10:46:13.005 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:13.094 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:13.221 INFO:teuthology.orchestra.run.smithi082.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T10:46:13.222 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:13.222 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T10:46:13.222 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:13.226 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:13.227 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:13.232 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:13.233 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:13.530 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:13.600 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:13.711 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:13.825 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:14.012 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:14.293 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:14.293 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:14.293 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:14.293 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:14.293 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:14.293 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:46:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:14.486 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:14.486 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:14.486 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:14.500 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:46:14.700 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:46:15.071 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:46:15.228 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 145 k 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 436 k 2024-03-29T10:46:15.229 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:15.265 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:15.419 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 145 k 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 436 k 2024-03-29T10:46:15.420 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:15.454 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 648 kB/s | 145 kB 00:00 2024-03-29T10:46:15.455 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:15.455 INFO:teuthology.orchestra.run.smithi079.stdout:Total 644 kB/s | 145 kB 00:00 2024-03-29T10:46:15.455 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:15.462 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:15.462 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:15.514 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:15.514 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:15.650 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:15.697 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:46:15.722 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:46:15.722 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:15.722 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:46:15.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:15.727 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 474 kB/s | 145 kB 00:00 2024-03-29T10:46:15.727 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:15.727 INFO:teuthology.orchestra.run.smithi082.stdout:Total 472 kB/s | 145 kB 00:00 2024-03-29T10:46:15.728 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:15.735 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:15.735 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:15.787 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:15.787 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:15.921 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:46:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:46:16.003 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:16.003 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:46:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:16.731 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:46:16.731 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:16.731 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:16.731 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:16.731 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:16.731 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:16.966 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-03-29T10:46:17.005 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:46:17.005 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:17.005 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:17.005 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:17.005 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:17.006 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:17.279 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr 2024-03-29T10:46:17.539 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:17.638 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:17.689 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:17.690 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:17.690 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:17.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:46:17.846 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:17.948 INFO:teuthology.orchestra.run.smithi082.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:18.001 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:18.002 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:18.002 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:18.065 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:46:18.329 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:18.477 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:18.478 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-03-29T10:46:18.478 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:18.478 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-03-29T10:46:18.478 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:46:18.479 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout:====================================================================================== 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.9 M 2024-03-29T10:46:18.480 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-03-29T10:46:18.481 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:18.625 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:18.775 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout:====================================================================================== 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout:====================================================================================== 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:46:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout:====================================================================================== 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-03-29T10:46:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:18.779 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 6.9 M 2024-03-29T10:46:18.779 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-03-29T10:46:18.779 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:19.114 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 167 kB/s | 31 kB 00:00 2024-03-29T10:46:19.131 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 75 kB/s | 15 kB 00:00 2024-03-29T10:46:19.168 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 162 kB/s | 31 kB 00:00 2024-03-29T10:46:19.194 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 70 kB/s | 15 kB 00:00 2024-03-29T10:46:19.298 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-29T10:46:19.332 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T10:46:19.354 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T10:46:19.386 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T10:46:19.428 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.9 MB/s | 1.8 MB 00:00 2024-03-29T10:46:19.457 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.4 MB/s | 1.8 MB 00:00 2024-03-29T10:46:19.574 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.1 MB/s | 863 kB 00:00 2024-03-29T10:46:19.653 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-29T10:46:19.683 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 87 kB/s | 19 kB 00:00 2024-03-29T10:46:19.783 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T10:46:19.804 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T10:46:19.841 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 529 kB/s | 31 kB 00:00 2024-03-29T10:46:19.867 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 786 kB/s | 144 kB 00:00 2024-03-29T10:46:19.879 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-03-29T10:46:19.917 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-03-29T10:46:19.946 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T10:46:20.005 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-03-29T10:46:20.030 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T10:46:20.092 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.9 MB/s | 2.0 MB 00:00 2024-03-29T10:46:20.097 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:20.097 INFO:teuthology.orchestra.run.smithi082.stdout:Total 5.2 MB/s | 6.9 MB 00:01 2024-03-29T10:46:20.115 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:20.132 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:20.132 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:20.139 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-03-29T10:46:20.144 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:20.144 INFO:teuthology.orchestra.run.smithi079.stdout:Total 4.1 MB/s | 6.9 MB 00:01 2024-03-29T10:46:20.162 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:20.179 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:20.179 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:20.294 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:20.295 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:20.354 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:20.354 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:20.811 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:20.863 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:20.884 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:46:20.919 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:46:20.939 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:46:20.966 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:46:20.983 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:46:21.005 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:46:21.030 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:46:21.060 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:46:21.110 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:46:21.169 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:46:21.238 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:46:21.293 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:46:21.333 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:46:21.376 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:46:21.405 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:46:21.407 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:46:21.437 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:46:21.457 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:46:21.486 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:46:21.518 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:46:21.677 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:46:21.712 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:46:21.759 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:46:21.793 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:46:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:46:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:46:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:46:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:46:22.514 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:46:22.515 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:46:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:46:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:46:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:46:22.595 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:46:22.596 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:46:22.858 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:46:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:46:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:46:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:46:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:46:22.859 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:22.859 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:46:22.966 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:22.967 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:23.101 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:46:23.211 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:46:23.658 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:23.772 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:23.810 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:46:23.811 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout:Install 12 Packages 2024-03-29T10:46:23.812 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:23.813 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 37 M 2024-03-29T10:46:23.813 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 209 M 2024-03-29T10:46:23.813 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:23.925 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:46:23.926 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout:============================================================================================ 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-03-29T10:46:23.927 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:23.928 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 37 M 2024-03-29T10:46:23.928 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-03-29T10:46:23.928 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:24.091 INFO:teuthology.orchestra.run.smithi082.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 251 kB/s | 32 kB 00:00 2024-03-29T10:46:24.124 INFO:teuthology.orchestra.run.smithi082.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-29T10:46:24.192 INFO:teuthology.orchestra.run.smithi082.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T10:46:24.225 INFO:teuthology.orchestra.run.smithi082.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 347 kB/s | 34 kB 00:00 2024-03-29T10:46:24.400 INFO:teuthology.orchestra.run.smithi082.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-29T10:46:24.467 INFO:teuthology.orchestra.run.smithi082.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:46:24.481 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 32 kB 00:00 2024-03-29T10:46:24.501 INFO:teuthology.orchestra.run.smithi082.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:46:24.531 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 350 kB/s | 17 kB 00:00 2024-03-29T10:46:24.582 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-29T10:46:24.726 INFO:teuthology.orchestra.run.smithi082.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 9.7 MB/s | 7.4 MB 00:00 2024-03-29T10:46:24.732 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.4 MB/s | 3.0 MB 00:00 2024-03-29T10:46:24.885 INFO:teuthology.orchestra.run.smithi082.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T10:46:24.994 INFO:teuthology.orchestra.run.smithi082.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-29T10:46:25.024 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-29T10:46:25.085 INFO:teuthology.orchestra.run.smithi082.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 548 kB/s | 195 kB 00:00 2024-03-29T10:46:25.091 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-29T10:46:25.110 INFO:teuthology.orchestra.run.smithi082.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T10:46:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:25.112 INFO:teuthology.orchestra.run.smithi082.stdout:Total 29 MB/s | 37 MB 00:01 2024-03-29T10:46:25.116 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:46:25.152 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:25.175 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:25.175 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:25.283 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-29T10:46:25.442 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T10:46:25.467 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T10:46:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:25.538 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:25.617 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T10:46:25.726 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.3 MB/s | 7.4 MB 00:01 2024-03-29T10:46:25.728 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:25.729 INFO:teuthology.orchestra.run.smithi079.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T10:46:25.769 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:25.794 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:25.794 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:26.148 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:26.148 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:26.253 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:26.348 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:46:26.381 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:46:26.475 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:46:26.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:46:26.687 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:46:26.717 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:46:26.793 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:46:26.859 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:26.932 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:46:26.964 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:46:27.057 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:46:27.109 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:46:27.286 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:46:27.317 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:46:27.393 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:46:27.453 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:46:27.563 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:46:28.039 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:46:28.166 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:46:28.805 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:46:29.411 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:46:29.828 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:46:29.859 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:46:30.451 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:46:30.479 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:46:32.401 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:46:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:46:32.402 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:46:32.811 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:46:32.811 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:32.811 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:32.812 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:33.211 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:46:33.240 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:46:33.240 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:46:33.240 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:46:33.240 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:46:33.240 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:46:33.241 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:46:33.720 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:46:33.721 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:46:33.722 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:46:33.722 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:46:33.722 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:33.722 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:33.781 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:33.931 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:46:33.932 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:46:33.933 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:33.934 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:33.934 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T10:46:33.934 INFO:teuthology.orchestra.run.smithi082.stdout:Install 13 Packages 2024-03-29T10:46:33.934 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:33.935 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 2.0 M 2024-03-29T10:46:33.935 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 26 M 2024-03-29T10:46:33.935 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:34.084 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:46:34.110 INFO:teuthology.orchestra.run.smithi082.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 145 kB/s | 18 kB 00:00 2024-03-29T10:46:34.152 INFO:teuthology.orchestra.run.smithi082.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 146 kB/s | 24 kB 00:00 2024-03-29T10:46:34.194 INFO:teuthology.orchestra.run.smithi082.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 250 kB/s | 52 kB 00:00 2024-03-29T10:46:34.228 INFO:teuthology.orchestra.run.smithi082.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-29T10:46:34.261 INFO:teuthology.orchestra.run.smithi082.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-29T10:46:34.287 INFO:teuthology.orchestra.run.smithi082.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2024-03-29T10:46:34.313 INFO:teuthology.orchestra.run.smithi082.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-03-29T10:46:34.652 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:34.662 INFO:teuthology.orchestra.run.smithi082.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 40 kB/s | 15 kB 00:00 2024-03-29T10:46:34.679 INFO:teuthology.orchestra.run.smithi082.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 77 kB/s | 32 kB 00:00 2024-03-29T10:46:34.805 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:34.806 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T10:46:34.806 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:34.806 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T10:46:34.806 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:46:34.807 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:34.808 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.0 M 2024-03-29T10:46:34.809 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-03-29T10:46:34.809 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:34.879 INFO:teuthology.orchestra.run.smithi082.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-03-29T10:46:34.921 INFO:teuthology.orchestra.run.smithi082.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 333 kB/s | 203 kB 00:00 2024-03-29T10:46:35.008 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 198 kB/s | 24 kB 00:00 2024-03-29T10:46:35.032 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-29T10:46:35.038 INFO:teuthology.orchestra.run.smithi082.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-29T10:46:35.083 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:46:35.108 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:46:35.134 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:46:35.159 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 851 kB/s | 43 kB 00:00 2024-03-29T10:46:35.184 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 174 kB/s | 52 kB 00:00 2024-03-29T10:46:35.230 INFO:teuthology.orchestra.run.smithi082.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-29T10:46:35.234 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-29T10:46:35.235 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:35.235 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T10:46:35.235 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:35.249 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:35.249 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:35.251 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-03-29T10:46:35.318 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-29T10:46:35.342 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:35.344 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-29T10:46:35.344 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:35.369 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-29T10:46:35.536 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-03-29T10:46:35.537 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:35.538 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T10:46:35.538 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:35.551 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:35.552 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:35.643 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:35.643 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:35.848 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:36.025 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:46:36.081 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:46:36.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:46:36.162 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:36.191 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:46:36.232 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:46:36.332 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:46:36.333 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:46:36.387 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:46:36.434 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:46:36.451 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:46:36.488 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:46:36.491 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:46:36.530 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:46:36.624 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:46:36.741 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:46:36.798 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:46:37.207 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:46:37.247 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:46:37.281 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:46:37.329 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:46:37.351 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:46:37.524 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:46:37.571 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:46:37.605 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:46:37.653 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:46:37.675 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:46:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:46:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:46:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:46:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:46:38.087 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:46:38.088 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:46:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:46:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:46:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:46:38.444 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:46:38.445 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:46:38.457 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:46:38.457 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:38.457 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:38.457 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:46:38.458 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:46:38.459 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:46:38.459 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:46:38.459 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:38.459 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:38.702 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:46:38.787 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:38.788 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:39.057 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:46:39.270 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:39.421 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:46:39.422 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================== 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout:Install 6 Packages 2024-03-29T10:46:39.423 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:39.424 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 6.8 M 2024-03-29T10:46:39.424 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 33 M 2024-03-29T10:46:39.424 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:39.625 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:39.778 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:39.779 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T10:46:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:39.779 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T10:46:39.779 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:39.779 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-03-29T10:46:39.780 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:39.781 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.8 M 2024-03-29T10:46:39.781 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-03-29T10:46:39.781 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:39.782 INFO:teuthology.orchestra.run.smithi082.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-29T10:46:39.857 INFO:teuthology.orchestra.run.smithi082.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:46:39.974 INFO:teuthology.orchestra.run.smithi082.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 358 kB/s | 138 kB 00:00 2024-03-29T10:46:40.208 INFO:teuthology.orchestra.run.smithi082.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-29T10:46:40.224 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:46:40.249 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 451 kB/s | 138 kB 00:00 2024-03-29T10:46:40.316 INFO:teuthology.orchestra.run.smithi082.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-29T10:46:40.374 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 461 kB/s | 58 kB 00:00 2024-03-29T10:46:40.383 INFO:teuthology.orchestra.run.smithi082.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 916 kB/s | 548 kB 00:00 2024-03-29T10:46:40.385 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:40.385 INFO:teuthology.orchestra.run.smithi082.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-03-29T10:46:40.399 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:40.409 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:40.409 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:40.425 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-29T10:46:40.475 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T10:46:40.497 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:40.498 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:40.500 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 445 kB/s | 248 kB 00:00 2024-03-29T10:46:40.501 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:40.501 INFO:teuthology.orchestra.run.smithi079.stdout:Total 9.4 MB/s | 6.8 MB 00:00 2024-03-29T10:46:40.515 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:40.525 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:40.525 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:40.613 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:40.613 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:40.802 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:40.907 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:40.966 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:46:41.018 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:46:41.074 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:46:41.144 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:46:41.537 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:46:41.630 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:46:41.672 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:46:41.693 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:46:41.716 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:46:41.754 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:46:41.816 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:46:41.840 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:46:42.575 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:46:42.576 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:46:42.576 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:46:42.576 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:46:42.576 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:46:42.576 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:46:42.709 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:46:42.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:46:42.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:46:42.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:46:42.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:46:42.709 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:46:42.879 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:46:42.880 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:46:42.880 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:42.880 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:42.992 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:43.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-fuse 2024-03-29T10:46:43.247 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-03-29T10:46:43.666 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:43.805 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:43.814 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:43.814 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-03-29T10:46:43.815 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:43.816 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 964 k 2024-03-29T10:46:43.816 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 2.8 M 2024-03-29T10:46:43.816 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:43.937 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 653 kB/s | 79 kB 00:00 2024-03-29T10:46:43.953 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:46:43.954 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 964 k 2024-03-29T10:46:43.955 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.8 M 2024-03-29T10:46:43.956 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:44.087 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 600 kB/s | 79 kB 00:00 2024-03-29T10:46:44.270 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T10:46:44.272 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:44.272 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T10:46:44.272 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:44.292 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:44.292 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:44.365 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:44.365 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:44.428 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 885 kB 00:00 2024-03-29T10:46:44.429 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:44.430 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-29T10:46:44.430 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:44.450 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:44.450 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:44.523 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:46:44.955 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:46:45.442 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:46:45.443 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:46:45.625 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:46:45.625 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:46:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:46:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:45.738 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:46:45.738 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:45.738 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:45.976 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:46:45.976 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:45.976 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:45.976 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:46:45.976 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:45.976 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:46.061 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-volume 2024-03-29T10:46:46.272 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2024-03-29T10:46:46.641 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:46.798 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:46.798 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout:Install 2 Packages 2024-03-29T10:46:46.799 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:46.800 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 332 k 2024-03-29T10:46:46.800 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 1.5 M 2024-03-29T10:46:46.800 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:46.849 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:46.934 INFO:teuthology.orchestra.run.smithi082.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 514 kB/s | 69 kB 00:00 2024-03-29T10:46:47.000 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:46:47.001 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 332 k 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.5 M 2024-03-29T10:46:47.002 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:47.127 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 554 kB/s | 69 kB 00:00 2024-03-29T10:46:47.209 INFO:teuthology.orchestra.run.smithi082.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 645 kB/s | 264 kB 00:00 2024-03-29T10:46:47.210 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:47.210 INFO:teuthology.orchestra.run.smithi082.stdout:Total 809 kB/s | 332 kB 00:00 2024-03-29T10:46:47.210 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:47.219 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:47.219 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:47.255 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:47.255 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:47.402 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:47.410 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 647 kB/s | 264 kB 00:00 2024-03-29T10:46:47.411 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:47.412 INFO:teuthology.orchestra.run.smithi079.stdout:Total 811 kB/s | 332 kB 00:00 2024-03-29T10:46:47.412 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:47.420 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:47.420 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:47.456 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:47.457 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:47.565 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:46:47.611 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:46:47.622 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:46:47.628 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:47.628 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:46:47.628 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:47.816 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:46:47.855 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:46:47.871 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:46:47.871 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:46:47.871 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:46:47.871 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:48.489 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:46:48.607 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:48.827 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:48.968 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:46:48.968 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:48.968 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:48.968 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:46:48.969 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:46:48.969 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:48.969 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:49.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados-devel 2024-03-29T10:46:49.335 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-03-29T10:46:49.699 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:49.849 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:46:49.850 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:49.851 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 127 k 2024-03-29T10:46:49.851 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 456 k 2024-03-29T10:46:49.851 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:49.908 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:50.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 127 k 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 456 k 2024-03-29T10:46:50.059 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:50.082 INFO:teuthology.orchestra.run.smithi082.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 549 kB/s | 127 kB 00:00 2024-03-29T10:46:50.082 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:50.083 INFO:teuthology.orchestra.run.smithi082.stdout:Total 546 kB/s | 127 kB 00:00 2024-03-29T10:46:50.083 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:50.088 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:50.088 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:50.143 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:50.144 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:50.288 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:50.293 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 543 kB/s | 127 kB 00:00 2024-03-29T10:46:50.294 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:50.294 INFO:teuthology.orchestra.run.smithi079.stdout:Total 539 kB/s | 127 kB 00:00 2024-03-29T10:46:50.294 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:50.299 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:50.300 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:50.355 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:50.355 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:50.396 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:50.500 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:50.608 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:50.833 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:51.086 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:51.142 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:51.142 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:51.142 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:51.142 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:51.142 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:51.142 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:51.316 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:51.316 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:51.316 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:51.316 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:51.316 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:51.316 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:51.392 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs2 2024-03-29T10:46:51.579 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-03-29T10:46:51.967 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:52.068 INFO:teuthology.orchestra.run.smithi082.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:52.124 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:52.125 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:52.125 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:52.153 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:52.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install libcephfs-devel 2024-03-29T10:46:52.256 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:52.311 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:52.312 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:52.312 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:52.393 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-03-29T10:46:52.752 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:52.903 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:46:52.904 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:52.905 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 31 k 2024-03-29T10:46:52.905 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 140 k 2024-03-29T10:46:52.905 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:52.953 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:53.103 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:53.104 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:53.104 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T10:46:53.104 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:53.104 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 31 k 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 140 k 2024-03-29T10:46:53.105 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:46:53.319 INFO:teuthology.orchestra.run.smithi082.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 75 kB/s | 31 kB 00:00 2024-03-29T10:46:53.319 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:53.320 INFO:teuthology.orchestra.run.smithi082.stdout:Total 75 kB/s | 31 kB 00:00 2024-03-29T10:46:53.320 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:46:53.323 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:46:53.323 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:46:53.354 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:46:53.354 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:46:53.423 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 98 kB/s | 31 kB 00:00 2024-03-29T10:46:53.424 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:46:53.424 INFO:teuthology.orchestra.run.smithi079.stdout:Total 97 kB/s | 31 kB 00:00 2024-03-29T10:46:53.424 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:46:53.427 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:46:53.428 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:46:53.449 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:46:53.450 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:46:53.482 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:46:53.585 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:46:53.591 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:53.688 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:53.961 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:54.101 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:54.225 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:54.225 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:54.225 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:46:54.225 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:54.225 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:54.225 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:54.374 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:46:54.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:54.375 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:46:54.375 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:46:54.375 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:54.375 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:54.382 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librados2 2024-03-29T10:46:54.550 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-03-29T10:46:54.942 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:55.039 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:55.090 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:55.091 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:55.091 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:55.107 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:55.137 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install librbd1 2024-03-29T10:46:55.203 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:55.255 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:55.256 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:55.256 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:55.311 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-03-29T10:46:55.700 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:55.797 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:55.849 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:55.850 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:55.850 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:55.868 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:55.900 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rados 2024-03-29T10:46:55.966 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:56.016 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:56.017 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:56.018 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:56.078 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-03-29T10:46:56.465 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:56.563 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:56.615 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:56.616 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:56.616 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:56.633 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:56.667 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rgw 2024-03-29T10:46:56.729 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:56.781 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:56.782 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:56.782 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:56.832 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-03-29T10:46:57.230 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:57.328 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:57.380 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:57.381 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:57.381 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:57.394 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:57.429 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-cephfs 2024-03-29T10:46:57.492 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:57.545 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:57.546 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:57.546 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:57.590 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-03-29T10:46:57.994 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:58.092 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:58.144 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:58.145 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:58.145 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:58.153 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:58.194 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install python3-rbd 2024-03-29T10:46:58.253 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:58.304 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:58.305 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:58.305 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:58.366 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-03-29T10:46:58.757 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:58.855 INFO:teuthology.orchestra.run.smithi082.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:58.907 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:58.908 INFO:teuthology.orchestra.run.smithi082.stdout:Nothing to do. 2024-03-29T10:46:58.908 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:46:58.928 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:58.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-fuse 2024-03-29T10:46:59.027 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:46:59.078 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:59.080 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-03-29T10:46:59.080 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:46:59.137 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-03-29T10:46:59.512 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:59.660 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:46:59.661 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:59.661 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-03-29T10:46:59.661 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:59.661 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 85 k 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 231 k 2024-03-29T10:46:59.662 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:46:59.696 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:46:59.843 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:59.844 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:46:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:46:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:46:59.845 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:46:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 85 k 2024-03-29T10:46:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 231 k 2024-03-29T10:46:59.845 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:47:00.080 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 204 kB/s | 85 kB 00:00 2024-03-29T10:47:00.080 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:00.080 INFO:teuthology.orchestra.run.smithi082.stdout:Total 204 kB/s | 85 kB 00:00 2024-03-29T10:47:00.080 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:47:00.087 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:47:00.088 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:47:00.144 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:47:00.144 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:47:00.284 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 194 kB/s | 85 kB 00:00 2024-03-29T10:47:00.285 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:00.285 INFO:teuthology.orchestra.run.smithi079.stdout:Total 193 kB/s | 85 kB 00:00 2024-03-29T10:47:00.286 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:47:00.291 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:47:00.293 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:47:00.293 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:47:00.348 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:47:00.349 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:47:00.395 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:00.484 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:47:00.599 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:00.798 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:01.028 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:01.150 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:47:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:01.150 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:01.150 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:47:01.305 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:01.306 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:01.306 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:47:01.306 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:01.306 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:01.306 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:47:01.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-mirror 2024-03-29T10:47:01.528 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-03-29T10:47:01.951 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:47:02.115 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:47:02.116 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:47:02.116 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repo Size 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 3.1 M 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 13 M 2024-03-29T10:47:02.117 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:47:02.128 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:47:02.288 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:47:02.288 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:47:02.288 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-03-29T10:47:02.288 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:47:02.288 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.1 M 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 13 M 2024-03-29T10:47:02.289 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:47:02.957 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.6 MB/s | 3.1 MB 00:00 2024-03-29T10:47:02.957 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:02.958 INFO:teuthology.orchestra.run.smithi079.stdout:Total 4.6 MB/s | 3.1 MB 00:00 2024-03-29T10:47:02.958 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:47:02.967 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:47:02.967 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:47:03.040 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:47:03.040 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:47:03.358 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.5 MB/s | 3.1 MB 00:01 2024-03-29T10:47:03.359 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:03.359 INFO:teuthology.orchestra.run.smithi082.stdout:Total 2.5 MB/s | 3.1 MB 00:01 2024-03-29T10:47:03.360 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:47:03.370 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:47:03.370 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:47:03.446 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:47:03.446 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:47:03.464 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:47:03.516 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:03.541 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:47:03.541 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:47:03.541 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:47:03.541 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:47:03.541 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:03.871 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:47:03.926 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:03.950 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:03.951 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:47:03.951 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:47:03.951 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:47:03.951 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:47:03.951 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:04.616 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:04.616 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:04.616 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:47:04.616 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:04.616 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:04.616 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:47:04.867 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-03-29T10:47:05.107 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:05.107 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:05.107 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:47:05.107 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:05.107 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:05.107 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:47:05.364 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install rbd-nbd 2024-03-29T10:47:05.440 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:47:05.592 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-03-29T10:47:05.592 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:47:05.592 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-03-29T10:47:05.592 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:47:05.592 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-03-29T10:47:05.593 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:47:05.593 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:05.593 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-03-29T10:47:05.593 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-03-29T10:47:05.593 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-03-29T10:47:05.593 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:05.594 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 171 k 2024-03-29T10:47:05.594 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 494 k 2024-03-29T10:47:05.594 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-03-29T10:47:05.904 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 553 kB/s | 171 kB 00:00 2024-03-29T10:47:05.904 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:05.904 INFO:teuthology.orchestra.run.smithi079.stdout:Total 550 kB/s | 171 kB 00:00 2024-03-29T10:47:05.905 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-03-29T10:47:05.912 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-03-29T10:47:05.913 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-03-29T10:47:05.941 INFO:teuthology.orchestra.run.smithi082.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:45:17 AM UTC. 2024-03-29T10:47:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-03-29T10:47:05.969 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-03-29T10:47:06.089 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout: Package Architecture Version Repository Size 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-29T10:47:06.090 INFO:teuthology.orchestra.run.smithi082.stdout:================================================================================ 2024-03-29T10:47:06.091 INFO:teuthology.orchestra.run.smithi082.stdout:Install 1 Package 2024-03-29T10:47:06.091 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:06.091 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 171 k 2024-03-29T10:47:06.091 INFO:teuthology.orchestra.run.smithi082.stdout:Installed size: 494 k 2024-03-29T10:47:06.091 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-29T10:47:06.127 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-03-29T10:47:06.219 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:06.303 INFO:teuthology.orchestra.run.smithi082.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 809 kB/s | 171 kB 00:00 2024-03-29T10:47:06.303 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:47:06.303 INFO:teuthology.orchestra.run.smithi082.stdout:Total 804 kB/s | 171 kB 00:00 2024-03-29T10:47:06.304 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-29T10:47:06.312 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-29T10:47:06.312 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-29T10:47:06.367 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-29T10:47:06.367 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-29T10:47:06.509 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-29T10:47:06.618 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:06.705 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:07.003 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-03-29T10:47:07.003 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:07.004 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:07.004 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-03-29T10:47:07.016 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:07.215 DEBUG:teuthology.parallel:result is None 2024-03-29T10:47:07.297 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:47:07.297 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:07.298 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-29T10:47:07.298 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:47:07.298 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:07.298 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-29T10:47:07.521 DEBUG:teuthology.parallel:result is None 2024-03-29T10:47:07.522 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:47:07.736 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:47:07.760 INFO:teuthology.orchestra.run.smithi079.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:07.760 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:07.760 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:47:07.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:47:07.899 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:47:07.920 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:07.920 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:47:07.920 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:47:07.922 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:47:07.922 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:07.922 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:47:07.953 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:07.953 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:47:07.987 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:47:07.988 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:07.988 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:47:08.023 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:47:08.094 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:08.094 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:47:08.121 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:47:08.188 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:47:08.188 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:08.188 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:47:08.217 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:47:08.285 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:08.285 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:47:08.312 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:47:08.379 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:47:08.379 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:08.379 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:47:08.408 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:47:08.475 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:08.475 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:47:08.501 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:47:08.569 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:47:08.651 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:47:08.652 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /var/log/ceph 2024-03-29T10:47:08.654 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2024-03-29T10:47:08.680 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:47:08.681 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:47:08.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:47:08.748 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:47:08.748 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:47:08.788 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:47:08.825 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:47:08.825 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:47:08.825 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622651', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi079.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPN+qhibP5MMVarEo4s+gy8Sm4/97TldBKn8eK5G+fp0IKtrXNwB1gabHb7cTMmUTPY2wNZf3cU4EgfTWGG5M34=', 'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLn8du5iTM/9h2XN3fmByzGpl7IBQMOUJR3iOwvLiEQM1xEK7UgdfKPxV2SN95N9JoBj4dZl5JKZauekhccJMRE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:47:08.825 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:47:08.857 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:47:08.871 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:47:08.911 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:47:08.939 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:08.939 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:47:08.991 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:47:08.991 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 10:47:03.804884866 +0000 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 10:44:13.713764538 +0000 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 10:44:13.713764538 +0000 2024-03-29T10:47:09.046 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 10:44:13.713764538 +0000 2024-03-29T10:47:09.047 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:47:09.111 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T10:47:09.111 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T10:47:09.111 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000259288 s, 2.0 MB/s 2024-03-29T10:47:09.112 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:47:09.168 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-03-29T10:47:09.223 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:47:09.223 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 10:47:03.804884866 +0000 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 10:44:14.057756703 +0000 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 10:44:14.057756703 +0000 2024-03-29T10:47:09.224 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 10:44:14.057756703 +0000 2024-03-29T10:47:09.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:47:09.289 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T10:47:09.289 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T10:47:09.289 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000186476 s, 2.7 MB/s 2024-03-29T10:47:09.290 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:47:09.346 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 10:47:03.805884843 +0000 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 10:44:14.396748981 +0000 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 10:44:14.396748981 +0000 2024-03-29T10:47:09.402 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 10:44:14.396748981 +0000 2024-03-29T10:47:09.403 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:47:09.467 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T10:47:09.467 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T10:47:09.467 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.00016268 s, 3.1 MB/s 2024-03-29T10:47:09.468 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:47:09.523 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-03-29T10:47:09.577 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-03-29 10:47:03.805884843 +0000 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-03-29 10:44:14.746741009 +0000 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-03-29 10:44:14.746741009 +0000 2024-03-29T10:47:09.578 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-03-29 10:44:14.746741009 +0000 2024-03-29T10:47:09.578 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:47:09.642 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-03-29T10:47:09.643 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-03-29T10:47:09.643 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000177592 s, 2.9 MB/s 2024-03-29T10:47:09.644 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:47:09.699 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:09.699 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:09.700 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:47:09.718 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:47:09.718 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 10:47:04.231349299 +0000 2024-03-29T10:47:09.776 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 10:43:11.057118213 +0000 2024-03-29T10:47:09.777 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 10:43:11.057118213 +0000 2024-03-29T10:47:09.777 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 10:43:11.057118213 +0000 2024-03-29T10:47:09.777 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:47:09.846 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T10:47:09.846 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T10:47:09.846 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000223166 s, 2.3 MB/s 2024-03-29T10:47:09.847 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:47:09.905 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 10:47:04.231349299 +0000 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 10:43:11.402112194 +0000 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 10:43:11.402112194 +0000 2024-03-29T10:47:09.963 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 10:43:11.402112194 +0000 2024-03-29T10:47:09.964 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:47:10.031 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T10:47:10.031 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T10:47:10.031 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000236852 s, 2.2 MB/s 2024-03-29T10:47:10.032 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:47:10.089 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 10:47:04.231349299 +0000 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 10:43:11.775105687 +0000 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 10:43:11.775105687 +0000 2024-03-29T10:47:10.145 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 10:43:11.775105687 +0000 2024-03-29T10:47:10.146 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:47:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T10:47:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T10:47:10.212 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000196714 s, 2.6 MB/s 2024-03-29T10:47:10.213 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:47:10.269 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-03-29T10:47:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:47:10.324 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:47:10.324 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T10:47:10.324 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:47:10.324 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:47:10.324 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-29 10:47:04.232349281 +0000 2024-03-29T10:47:10.325 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-29 10:43:12.112099807 +0000 2024-03-29T10:47:10.325 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-29 10:43:12.112099807 +0000 2024-03-29T10:47:10.325 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-29 10:43:12.112099807 +0000 2024-03-29T10:47:10.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:47:10.390 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-29T10:47:10.390 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-29T10:47:10.390 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000194023 s, 2.6 MB/s 2024-03-29T10:47:10.392 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:47:10.451 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:10.451 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:47:10.452 INFO:tasks.ceph:Generating config... 2024-03-29T10:47:10.453 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:47:10.453 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:47:10.454 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:47:10.454 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:47:10.454 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:47:10.454 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:47:10.454 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:47:10.454 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:47:10.454 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:47:10.454 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:47:10.454 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:47:10.454 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:47:10.454 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:47:10.454 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:47:10.454 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:47:10.454 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:47:10.455 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:47:10.455 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:47:10.455 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:47:10.455 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:47:10.455 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:47:10.455 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:47:10.455 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:47:10.456 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:47:10.457 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:47:10.457 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:47:10.457 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:47:10.457 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:10.508 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:47:10.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:47:10.556 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:47:10.585 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.79:6789]'), ('mon.c', '[v1:172.21.15.79:6790]'), ('mon.b', '[v1:172.21.15.82:6789]')] 2024-03-29T10:47:10.586 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.79:6789],[v1:172.21.15.79:6790],[v1:172.21.15.82:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:47:10.587 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:10.587 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:47:10.641 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.79:6789]' --addv c '[v1:172.21.15.79:6790]' --addv b '[v1:172.21.15.82:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:10.726 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:10.726 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: generated fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:10.726 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = quincy 2024-03-29T10:47:10.726 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 0 2024-03-29T10:47:10.726 INFO:teuthology.orchestra.run.smithi079.stdout:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:10.726 INFO:teuthology.orchestra.run.smithi079.stdout:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:min_mon_release 17 (quincy) 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:election_strategy: 1 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:10.727 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:47:10.728 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:47:10.785 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 13c29107-6abd-40ad-99ae-e45192d5c573... 2024-03-29T10:47:10.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:47:10.827 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: chdir = "" 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: auth supported = cephx 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: filestore xattr use omap = true 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: mon clock drift allowed = 1.000 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.875 INFO:teuthology.orchestra.run.smithi079.stdout: osd crush chooseleaf type = 0 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: auth debug = true 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on old message = true 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: ms die on bug = true 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: mon pg warn max object skew = 0 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default size = 2 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow primary affinity = true 2024-03-29T10:47:10.876 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd allow pg remap = true 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on no sortbitwise = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on too few osds = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon_allow_pool_size_one = true 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: osd default data pool replay window = 5 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: mon allow pool delete = true 2024-03-29T10:47:10.877 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: mon cluster log file level = debug 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: debug asserts on shutdown = true 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: mon health detail to clog = false 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: mon host = "[v1:172.21.15.79:6789],[v1:172.21.15.79:6790],[v1:172.21.15.82:6789]" 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: lockdep = True 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: mon client directed command retry = 5 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: mon election default strategy = 1 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: ms bind msgr2 = False 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: ms inject socket failures = 5000 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: ms type = async 2024-03-29T10:47:10.878 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout:[osd] 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd journal size = 100 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub load threshold = 5.0 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd scrub max interval = 600 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd recover clone overlap = true 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd recovery max chunk = 1048576 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug shutdown = true 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug op order = true 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify stray on activate = true 2024-03-29T10:47:10.879 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug trim objects = true 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: osd open classes on start = true 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug pg log writeout = true 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: osd map max advance = 10 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: journal zero on create = true 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:47:10.880 INFO:teuthology.orchestra.run.smithi079.stdout: filestore apply finisher threads = 3 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: bdev debug aio = true 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug misdirected ops = true 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore block size = 96636764160 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression algorithm = zstd 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore compression mode = aggressive 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore fsck on mount = True 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: bluestore zero block detection = True 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluefs = 20 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: debug bluestore = 20 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T10:47:10.881 INFO:teuthology.orchestra.run.smithi079.stdout: debug osd = 20 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: debug rocksdb = 10 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd full ratio = 0.9 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd class default list = * 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd class load list = * 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd client watch timeout = 120 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify cached snaps = True 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd debug verify missing on start = True 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd mclock override recovery settings = True 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd objectstore = bluestore 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue = debug_random 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd op queue cut off = debug_random 2024-03-29T10:47:10.882 INFO:teuthology.orchestra.run.smithi079.stdout: osd shutdown pgref assert = True 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr] 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug mgr = 20 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: mgr/telemetry/nag = false 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout:[mon] 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug mon = 20 2024-03-29T10:47:10.883 INFO:teuthology.orchestra.run.smithi079.stdout: debug paxos = 20 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: debug auth = 20 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon data avail warn = 5 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon mgr mkfs grace = 240 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd reporter subtree level = osd 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon osd prime pg temp = true 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:47:10.884 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: # 1m isn't quite enough 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon min osdmap epochs = 50 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune interval = 2 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune min = 15 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon scrub interval = 300 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: mon warn on pool no app = False 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: paxos service trim min = 10 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout:[client] 2024-03-29T10:47:10.885 INFO:teuthology.orchestra.run.smithi079.stdout: rgw cache enabled = true 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable ops log = true 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: rgw enable usage log = true 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: debug monc = 20 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: debug ms = 1 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: debug objecter = 20 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout: debug rados = 20 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.a] 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.c] 2024-03-29T10:47:10.886 INFO:teuthology.orchestra.run.smithi079.stdout:[mon.b] 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2024-03-29T10:47:10.887 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:47:10.888 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2024-03-29T10:47:10.889 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = "[v1:172.21.15.79:6789],[v1:172.21.15.79:6790],[v1:172.21.15.82:6789]" 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: lockdep = True 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: mon client directed command retry = 5 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: mon election default strategy = 1 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: ms bind msgr2 = False 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject socket failures = 5000 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: ms type = async 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 600 2024-03-29T10:47:10.890 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug trim objects = true 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.891 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore block size = 96636764160 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore compression algorithm = zstd 2024-03-29T10:47:10.892 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore compression mode = aggressive 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore fsck on mount = True 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore zero block detection = True 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluefs = 20 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluestore = 20 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 20 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: debug rocksdb = 10 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd full ratio = 0.9 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: osd class default list = * 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: osd class load list = * 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: osd client watch timeout = 120 2024-03-29T10:47:10.893 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify cached snaps = True 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify missing on start = True 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock override recovery settings = True 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd objectstore = bluestore 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue = debug_random 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue cut off = debug_random 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: osd shutdown pgref assert = True 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:10.894 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.895 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon min osdmap epochs = 50 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune interval = 2 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune min = 15 2024-03-29T10:47:10.896 INFO:teuthology.orchestra.run.smithi082.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub interval = 300 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on pool no app = False 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: paxos service trim min = 10 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 20 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: debug objecter = 20 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout: debug rados = 20 2024-03-29T10:47:10.897 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2024-03-29T10:47:10.898 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.c] 2024-03-29T10:47:10.898 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.b] 2024-03-29T10:47:10.899 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:47:10.899 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:47:10.955 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:47:10.955 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:10.956 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:47:10.971 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:10.971 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:47:11.025 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:47:11.026 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:11.026 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:47:11.026 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:47:11.105 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:11.105 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:11.160 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:47:11.160 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:11.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:47:11.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:47:11.201 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:11.201 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:11.257 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:47:11.257 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:47:11.258 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:47:11.312 INFO:teuthology.orchestra.run.smithi079.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:47:11.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:47:11.400 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:47:11.402 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:47:11.402 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:47:11.402 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:47:11.444 INFO:teuthology.orchestra.run.smithi079.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:47:11.460 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:47:11.504 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:47:11.519 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:47:11.519 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi079.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:47:11.520 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:11.546 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:11.546 INFO:tasks.ceph:role: osd.0 2024-03-29T10:47:11.546 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:47:11.546 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:11.620 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:11.621 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:12.105 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T10:47:12.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T10:47:12.106 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:12.247 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:12.281 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:12.349 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:12.349 INFO:tasks.ceph:role: osd.1 2024-03-29T10:47:12.349 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:47:12.349 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:47:12.424 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:12.424 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:12.424 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:12.425 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:12.886 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T10:47:12.888 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T10:47:12.888 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:12.972 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:13.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:13.108 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:13.108 INFO:tasks.ceph:role: osd.2 2024-03-29T10:47:13.108 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:47:13.108 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:13.182 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:13.681 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T10:47:13.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T10:47:13.683 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:13.822 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:13.857 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:13.924 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:13.925 INFO:tasks.ceph:role: osd.3 2024-03-29T10:47:13.925 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:47:13.925 DEBUG:teuthology.orchestra.run.smithi079:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:47:13.999 INFO:teuthology.orchestra.run.smithi079.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:14.000 INFO:teuthology.orchestra.run.smithi079.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:14.481 INFO:teuthology.orchestra.run.smithi079.stdout:Discarding blocks...Done. 2024-03-29T10:47:14.483 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi079.front.sepia.ceph.com -o noatime 2024-03-29T10:47:14.484 DEBUG:teuthology.orchestra.run.smithi079:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:14.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:14.643 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:14.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:14.765+0000 7f3bbed49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:47:14.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:14.765+0000 7f3bbed49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:47:14.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:14.765+0000 7f3bbed49640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:47:14.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:14.765+0000 7f3bbed49640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:47:14.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:14.765+0000 7f3bbed49640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:47:14.749 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:14.766+0000 7f3bbed49640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:47:17.613 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:47:17.645 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:17.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:17.753+0000 7f2ea6cf9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:47:17.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:17.754+0000 7f2ea6cf9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:47:17.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:17.754+0000 7f2ea6cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:47:17.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:17.754+0000 7f2ea6cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:47:17.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:17.754+0000 7f2ea6cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:47:17.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:17.754+0000 7f2ea6cf9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:47:20.583 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:47:20.614 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:20.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:20.725+0000 7f0c90ad7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:47:20.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:20.725+0000 7f0c90ad7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:47:20.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:20.725+0000 7f0c90ad7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:47:20.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:20.725+0000 7f0c90ad7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:47:20.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:20.725+0000 7f0c90ad7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:47:20.709 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:20.726+0000 7f0c90ad7640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:47:23.561 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:47:23.592 DEBUG:teuthology.orchestra.run.smithi079:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:23.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:23.700+0000 7f1a5e749640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:47:23.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:23.701+0000 7f1a5e749640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:47:23.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:23.701+0000 7f1a5e749640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:47:23.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:23.701+0000 7f1a5e749640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:47:23.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:23.701+0000 7f1a5e749640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:47:23.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:23.701+0000 7f1a5e749640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:47:26.529 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:47:26.558 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:26.590 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:26.590 INFO:tasks.ceph:role: osd.4 2024-03-29T10:47:26.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:47:26.591 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:26.668 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:27.171 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T10:47:27.173 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T10:47:27.174 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:27.314 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:27.354 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:27.423 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:27.423 INFO:tasks.ceph:role: osd.5 2024-03-29T10:47:27.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:47:27.423 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:47:27.500 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:27.500 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:27.500 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:27.500 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:27.501 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:27.501 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:27.501 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:27.501 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:27.501 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:27.501 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:27.952 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T10:47:27.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T10:47:27.954 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:28.045 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:28.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:28.144 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:28.145 INFO:tasks.ceph:role: osd.6 2024-03-29T10:47:28.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:47:28.146 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:47:28.219 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:28.219 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:28.219 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:28.220 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:28.220 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:28.220 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:28.220 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:28.711 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T10:47:28.712 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T10:47:28.712 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:28.833 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:28.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:28.934 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:47:28.934 INFO:tasks.ceph:role: osd.7 2024-03-29T10:47:28.934 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:47:28.934 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:47:29.008 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:47:29.009 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:47:29.009 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:47:29.522 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-29T10:47:29.524 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-29T10:47:29.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:29.657 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:29.692 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:29.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:29.809+0000 7f75d100f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:47:29.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:29.809+0000 7f75d100f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:47:29.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:29.809+0000 7f75d100f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:47:29.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:29.809+0000 7f75d100f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:47:29.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:29.809+0000 7f75d100f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:47:29.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:29.809+0000 7f75d100f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:47:32.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:47:32.682 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:32.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:32.791+0000 7fe6cd1ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:47:32.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:32.791+0000 7fe6cd1ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:47:32.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:32.791+0000 7fe6cd1ac640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:47:32.775 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:32.791+0000 7fe6cd1ac640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:47:32.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:32.791+0000 7fe6cd1ac640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:47:32.776 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:32.791+0000 7fe6cd1ac640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:47:35.620 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:47:35.651 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:35.743 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:35.761+0000 7fa069011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:47:35.744 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:35.761+0000 7fa069011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:47:35.744 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:35.761+0000 7fa069011640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:47:35.744 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:35.761+0000 7fa069011640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:47:35.744 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:35.761+0000 7fa069011640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:47:35.744 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:35.762+0000 7fa069011640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:47:38.604 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:47:38.635 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:38.729 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:38.747+0000 7efda5012640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:47:38.729 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:38.747+0000 7efda5012640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:47:38.730 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:38.747+0000 7efda5012640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:47:38.730 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:38.747+0000 7efda5012640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:47:38.730 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:38.747+0000 7efda5012640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:47:38.730 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:38.747+0000 7efda5012640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:47:41.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:47:41.611 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:47:41.612 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:41.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:47:41.644 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:41.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:47:41.718 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:41.718 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:47:41.789 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:41.789 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:47:41.859 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:41.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:47:41.927 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:41.928 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:47:41.959 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:41.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:47:42.031 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:42.031 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:47:42.101 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:42.101 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:47:42.170 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:42.170 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:47:42.240 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:42.240 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:47:42.256 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:42.256 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:47:42.296 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:47:42.296 DEBUG:teuthology.orchestra.run.smithi079:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:47:42.298 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:47:42.339 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.y] 2024-03-29T10:47:42.339 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAvnAZmLIi3ExAAlTyNJlMMJUaEYfTj3vktCg== 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.0] 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAynAZmv/WuLRAA7lDRsH7SCBRXBvtP08HlIA== 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.1] 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQA1nAZmbXL8LBAAmkpcTpX5qj/vYF23ZU+blg== 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.2] 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQA4nAZm2ONLKxAA+wDBsMZFM9FzkmGGkBhePQ== 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.3] 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQA7nAZmM1/VKRAAB8hTsYboZG4l2HuqezPw2g== 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:[mgr.x] 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAvnAZmCpz1GBAAavL3YKq+jKysDwglWYG0EQ== 2024-03-29T10:47:42.340 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.4] 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBBnAZmYhVWMBAAxgz/J0+nJCEHoKy+ldcG5A== 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.5] 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBEnAZmReI4LxAAvSy5AQXgNgE+yYqnpZ3X0Q== 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.6] 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBHnAZmBtNsLRAABX+itrWJZK5bT20VseP+yA== 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout:[osd.7] 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQBKnAZmkEWVLBAAlpu3sb9nP06L7GfqfkQfWA== 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAvnAZmuHmgGxAAfs9MV1YyEkzQ3eQst+yZqA== 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout:[client.1] 2024-03-29T10:47:42.341 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQAvnAZm0HktHxAAb0xXizqttweMIm+OB6mu6A== 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.y] 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAvnAZmLIi3ExAAlTyNJlMMJUaEYfTj3vktCg== 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAynAZmv/WuLRAA7lDRsH7SCBRXBvtP08HlIA== 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQA1nAZmbXL8LBAAmkpcTpX5qj/vYF23ZU+blg== 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2024-03-29T10:47:42.362 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQA4nAZm2ONLKxAA+wDBsMZFM9FzkmGGkBhePQ== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQA7nAZmM1/VKRAAB8hTsYboZG4l2HuqezPw2g== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAvnAZmCpz1GBAAavL3YKq+jKysDwglWYG0EQ== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQBBnAZmYhVWMBAAxgz/J0+nJCEHoKy+ldcG5A== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQBEnAZmReI4LxAAvSy5AQXgNgE+yYqnpZ3X0Q== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.6] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQBHnAZmBtNsLRAABX+itrWJZK5bT20VseP+yA== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.7] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQBKnAZmkEWVLBAAlpu3sb9nP06L7GfqfkQfWA== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAvnAZmuHmgGxAAfs9MV1YyEkzQ3eQst+yZqA== 2024-03-29T10:47:42.363 INFO:teuthology.orchestra.run.smithi082.stdout:[client.1] 2024-03-29T10:47:42.364 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQAvnAZm0HktHxAAb0xXizqttweMIm+OB6mu6A== 2024-03-29T10:47:42.364 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:42.385 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:42.458 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.512 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.515 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.566 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.568 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.619 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.622 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.673 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:42.675 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:47:42.726 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.728 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.781 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.832 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.835 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.886 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.888 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:47:42.939 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:42.942 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:42.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:42.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:47:43.052 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:47:43.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:47:43.082 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:43.178 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:47:43.205 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:47:43.271 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:43.368 DEBUG:teuthology.orchestra.run.smithi079:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:47:43.395 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:47:43.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:47:43.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:47:43.546 DEBUG:teuthology.orchestra.run.smithi079:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:43.549 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:47:43.602 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:47:43.602 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:47:43.602 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:47:43.605 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:47:43.605 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:47:43.605 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:47:43.607 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:47:43.607 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:47:43.607 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:47:43.644 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:47:43.644 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:47:43.644 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:47:43.644 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:47:43.647 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:47:43.647 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:47:43.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:47:43.649 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:47:43.649 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:47:43.649 DEBUG:teuthology.orchestra.run.smithi079:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.766+0000 7fc1bdc4a640 10 monclient: get_monmap_and_config 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.766+0000 7fc1bdc4a640 10 monclient: build_initial_monmap 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.766+0000 7fc1bdc4a640 10 monclient: monmap: 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:43.768832+0000 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:43.768832+0000 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:43.750 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:43.751 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:43.751 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:43.751 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.767+0000 7fc1bdc4a640 1 Processor -- start 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.767+0000 7fc1bdc4a640 1 -- start start 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.767+0000 7fc1bdc4a640 10 monclient: init 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b80b8240 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:43.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b80b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b809c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b809bd80 con 0x7fc1b809c8c0 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b809b960 con 0x7fc1b80b1f60 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b80b2ec0 con 0x7fc1b80b8240 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1b7fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc1b809c8c0 0x7fc1b814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:43478/0 (socket says 172.21.15.79:43478) 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1b77fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fc1b80b8240 0x7fc1b813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:36390/0 (socket says 172.21.15.79:36390) 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1b7fff640 1 -- 172.21.15.79:0/944739914 learned_addr learned my addr 172.21.15.79:0/944739914 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:43.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:43.768+0000 7fc1bdc4a640 10 monclient(hunting): authenticate will time out at 983.061659s 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): tick 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 monclient(hunting): continuing hunt 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 -- 172.21.15.79:0/944739914 >> v1:172.21.15.79:6790/0 conn(0x7fc1b80b8240 legacy=0x7fc1b813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 -- 172.21.15.79:0/944739914 >> v1:172.21.15.82:6789/0 conn(0x7fc1b80b1f60 legacy=0x7fc1b809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 -- 172.21.15.79:0/944739914 >> v1:172.21.15.79:6789/0 conn(0x7fc1b809c8c0 legacy=0x7fc1b814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc1980036e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc1980072d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:46.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b809c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 -- 172.21.15.79:0/944739914 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b813fb60 con 0x7fc1b809c8c0 2024-03-29T10:47:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 -- 172.21.15.79:0/944739914 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19800f8b0 con 0x7fc1980072d0 2024-03-29T10:47:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 1 -- 172.21.15.79:0/944739914 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc198010b10 con 0x7fc1980036e0 2024-03-29T10:47:46.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:46.769+0000 7fc1b57fa640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:48.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.943+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191832031 0 0) 0x7fc1b813fb60 con 0x7fc1b809c8c0 2024-03-29T10:47:48.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.943+0000 7fc1b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.943+0000 7fc1b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.943+0000 7fc1b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.943+0000 7fc1b67fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:47:48.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.943+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a4003520 con 0x7fc1b809c8c0 2024-03-29T10:47:48.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 451968633 0 0) 0x7fc1a4003520 con 0x7fc1b809c8c0 2024-03-29T10:47:48.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 >> v1:172.21.15.79:6790/0 conn(0x7fc1980036e0 legacy=0x7fc198005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 >> v1:172.21.15.82:6789/0 conn(0x7fc1980072d0 legacy=0x7fc198009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 monclient: found mon.noname-a 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198011d70 con 0x7fc1b809c8c0 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc1b00019d0 con 0x7fc1b809c8c0 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: dump: 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:48.930 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1bdc4a640 5 monclient: authenticate success, global_id 4101 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1bdc4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1b0001ba0 con 0x7fc1b809c8c0 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 1 -- 172.21.15.79:0/944739914 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc1b0001f70 con 0x7fc1b809c8c0 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1bdc4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1bdc4a640 4 set_mon_vals no callback set 2024-03-29T10:47:48.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.946+0000 7fc1b67fc640 10 monclient: dump: 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 10 monclient: shutdown 2024-03-29T10:47:48.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:48.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/944739914 >> v1:172.21.15.79:6789/0 conn(0x7fc1b809c8c0 legacy=0x7fc19800d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/944739914 shutdown_connections 2024-03-29T10:47:48.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/944739914 >> 172.21.15.79:0/944739914 conn(0x7fc1b8066b00 msgr2=0x7fc1b8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:48.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/944739914 shutdown_connections 2024-03-29T10:47:48.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.947+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/944739914 wait complete. 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.948+0000 7fc1bdc4a640 10 monclient: build_initial_monmap 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.948+0000 7fc1bdc4a640 10 monclient: monmap: 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:48.937 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 1 librados: starting msgr at 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 1 librados: starting objecter 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 1 Processor -- start 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 1 -- start start 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 1 librados: setting wanted keys 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 1 librados: calling monclient init 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.949+0000 7fc1bdc4a640 10 monclient: init 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient(hunting): picked mon.noname-a con 0x7fc1980036e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:48.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient(hunting): picked mon.noname-c con 0x7fc1980072d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b809c8c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b80b1c10 con 0x7fc1980036e0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b81d7f80 con 0x7fc1b809c8c0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b81d7c50 con 0x7fc1980072d0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1b77fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc1980036e0 0x7fc1b81d7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:43534/0 (socket says 172.21.15.79:43534) 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1b77fe640 1 -- 172.21.15.79:0/1711156715 learned_addr learned my addr 172.21.15.79:0/1711156715 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 client.?.objecter ms_handle_connect 0x7fc1980036e0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 client.?.objecter ms_handle_connect 0x7fc1980072d0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 client.?.objecter ms_handle_connect 0x7fc1b809c8c0 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:48.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1bdc4a640 10 monclient(hunting): authenticate will time out at 988.242788s 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700730397 0 0) 0x7fc1b80b1c10 con 0x7fc1980036e0 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.950+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1980102e0 con 0x7fc1980036e0 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2081855919 0 0) 0x7fc1980102e0 con 0x7fc1980036e0 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b80b1c10 con 0x7fc1980036e0 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc1a0001880 con 0x7fc1980036e0 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient(hunting): dump: 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:48.940 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1024844700 0 0) 0x7fc1b80b1c10 con 0x7fc1980036e0 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 >> v1:172.21.15.79:6790/0 conn(0x7fc1980072d0 legacy=0x7fc1b80af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 >> v1:172.21.15.82:6789/0 conn(0x7fc1b809c8c0 legacy=0x7fc1b80b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 monclient: found mon.a 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8138f10 con 0x7fc1980036e0 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient: _finish_auth 0 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 10 monclient: _renew_subs 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b8139180 con 0x7fc1980036e0 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 10 monclient: _renew_subs 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b81e2fa0 con 0x7fc1980036e0 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.951+0000 7fc1bdc4a640 1 librados: init done 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a00047e0 con 0x7fc1980036e0 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1b5ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc1a0004af0 con 0x7fc1980036e0 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 10 monclient: dump: 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc1a0005060 con 0x7fc1980036e0 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 890746958 0 0) 0x7fc1a0006680 con 0x7fc1980036e0 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 10 client.4113.objecter ms_dispatch 0x7fc1b8000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.952+0000 7fc1affff640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.953+0000 7fc1bdc4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.953+0000 7fc1bdc4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.953+0000 7fc1bdc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.953+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c005080 con 0x7fc1980036e0 2024-03-29T10:47:48.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.955+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc1a0005cd0 con 0x7fc1980036e0 2024-03-29T10:47:48.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.955+0000 7fc1affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:48.955+0000 7fc1affff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:49.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.034+0000 7fc1bdc4a640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:47:49.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.034+0000 7fc1bdc4a640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:47:49.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.034+0000 7fc1bdc4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:49.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.034+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc19c005330 con 0x7fc1980036e0 2024-03-29T10:47:49.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.035+0000 7fc1affff640 1 -- 172.21.15.79:0/1711156715 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc1a00053f0 con 0x7fc1980036e0 2024-03-29T10:47:49.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.035+0000 7fc1affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:47:49.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.035+0000 7fc1affff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:47:49.017 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 10 librados: watch_flush enter 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 10 librados: watch_flush exit 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 10 monclient: shutdown 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 >> v1:172.21.15.79:6789/0 conn(0x7fc1980036e0 legacy=0x7fc1b81d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 shutdown_connections 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 >> 172.21.15.79:0/1711156715 conn(0x7fc1b8066b00 msgr2=0x7fc1b80b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 shutdown_connections 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 1 -- 172.21.15.79:0/1711156715 wait complete. 2024-03-29T10:47:49.019 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:49.036+0000 7fc1bdc4a640 1 librados: shutdown 2024-03-29T10:47:49.029 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:47:49.030 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:47:49.030 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:49.030 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:47:49.057 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:49.057 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:47:49.124 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:49.125 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:47:49.191 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-03-29T10:47:49.191 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:47:49.259 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:49.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:47:49.287 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:49.287 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:47:49.355 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:49.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:47:49.407 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:49.425+0000 7f793a01e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:47:49.409 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:49.427+0000 7feb80432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:47:49.422 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-29T10:47:49.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:47:49.462 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:49.480+0000 7feb80432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:47:49.463 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:49.481+0000 7f793a01e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:47:49.489 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new e014f81f-8d58-44ef-8d6e-2c0895a7c3bd 0 2024-03-29T10:47:49.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.646+0000 7efd83a55640 10 monclient: get_monmap_and_config 2024-03-29T10:47:49.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.646+0000 7efd83a55640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.629 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.647+0000 7efd83a55640 10 monclient: monmap: 2024-03-29T10:47:49.629 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:49.629 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:49.647850+0000 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:49.647850+0000 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.647+0000 7efd83a55640 1 Processor -- start 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.647+0000 7efd83a55640 1 -- start start 2024-03-29T10:47:49.630 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.647+0000 7efd83a55640 10 monclient: init 2024-03-29T10:47:49.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:49.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient(hunting): picked mon.noname-c con 0x7efd7c09ac40 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:49.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient(hunting): picked mon.noname-b con 0x7efd7c09c870 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:49.631 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient(hunting): picked mon.noname-a con 0x7efd7c0b4360 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:49.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c09dec0 con 0x7efd7c0b4360 2024-03-29T10:47:49.632 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c09da30 con 0x7efd7c09ac40 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0b39c0 con 0x7efd7c09c870 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd83a55640 10 monclient(hunting): authenticate will time out at 831.819621s 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd817f1640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7efd7c09ac40 0x7efd7c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57346/0 (socket says 172.21.15.82:57346) 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd817f1640 1 -- 172.21.15.82:0/1336587328 learned_addr learned my addr 172.21.15.82:0/1336587328 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426970721 0 0) 0x7efd7c09da30 con 0x7efd7c09ac40 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): my global_id is 4099 2024-03-29T10:47:49.640 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd68003520 con 0x7efd7c09ac40 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675069680 0 0) 0x7efd7c0b39c0 con 0x7efd7c09c870 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c09da30 con 0x7efd7c09c870 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960535050 0 0) 0x7efd74001490 con 0x7efd7c0b4360 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c0b39c0 con 0x7efd7c0b4360 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 209643026 0 0) 0x7efd68003520 con 0x7efd7c09ac40 2024-03-29T10:47:49.641 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 >> v1:172.21.15.79:6790/0 conn(0x7efd7c09c870 legacy=0x7efd7c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 >> v1:172.21.15.79:6789/0 conn(0x7efd7c0b4360 legacy=0x7efd7c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.648+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0b5160 con 0x7efd7c09ac40 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 5 monclient: authenticate success, global_id 4099 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7efd6c001690 con 0x7efd7c09ac40 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:49.642 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: dump: 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd6c001870 con 0x7efd7c09ac40 2024-03-29T10:47:49.643 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 1 -- 172.21.15.82:0/1336587328 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7efd6c001c80 con 0x7efd7c09ac40 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 4 set_mon_vals no callback set 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd73fff640 10 monclient: dump: 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.644 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 10 monclient: shutdown 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 1 -- 172.21.15.82:0/1336587328 >> v1:172.21.15.82:6789/0 conn(0x7efd7c09ac40 legacy=0x7efd7c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 1 -- 172.21.15.82:0/1336587328 shutdown_connections 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 1 -- 172.21.15.82:0/1336587328 >> 172.21.15.82:0/1336587328 conn(0x7efd7c066b00 msgr2=0x7efd7c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 1 -- 172.21.15.82:0/1336587328 shutdown_connections 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.649+0000 7efd83a55640 1 -- 172.21.15.82:0/1336587328 wait complete. 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.645 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 10 monclient: monmap: 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 1 librados: starting msgr at 2024-03-29T10:47:49.646 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 1 librados: starting objecter 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 1 Processor -- start 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 1 -- start start 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 1 librados: setting wanted keys 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 1 librados: calling monclient init 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.650+0000 7efd83a55640 10 monclient: init 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient(hunting): picked mon.noname-a con 0x7efd7c09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient(hunting): picked mon.noname-c con 0x7efd7c0acd00 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient(hunting): picked mon.noname-b con 0x7efd7c0af4c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0b12f0 con 0x7efd7c09ac40 2024-03-29T10:47:49.647 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c0b15d0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c1d70e0 con 0x7efd7c0acd00 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd83a55640 10 monclient(hunting): authenticate will time out at 831.822664s 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd81ff2640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7efd7c0af4c0 0x7efd7c139020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57352/0 (socket says 172.21.15.82:57352) 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd81ff2640 1 -- 172.21.15.82:0/2226433843 learned_addr learned my addr 172.21.15.82:0/2226433843 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 client.?.objecter ms_handle_connect 0x7efd7c0af4c0 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 client.?.objecter ms_handle_connect 0x7efd7c09ac40 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 client.?.objecter ms_handle_connect 0x7efd7c0acd00 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186933742 0 0) 0x7efd7c0b15d0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.648 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): my global_id is 4102 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd60004eb0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423492838 0 0) 0x7efd7c0b12f0 con 0x7efd7c09ac40 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c0b15d0 con 0x7efd7c09ac40 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969964931 0 0) 0x7efd7c1d70e0 con 0x7efd7c0acd00 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.649 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c0b12f0 con 0x7efd7c0acd00 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628751236 0 0) 0x7efd60004eb0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd7c1d70e0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7efd74001bc0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.651+0000 7efd71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.650 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172637688 0 0) 0x7efd7c0b15d0 con 0x7efd7c09ac40 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd60004eb0 con 0x7efd7c09ac40 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7efd6c001b80 con 0x7efd7c09ac40 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.651 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3296662644 0 0) 0x7efd7c0b12f0 con 0x7efd7c0acd00 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd7c0b15d0 con 0x7efd7c0acd00 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7efd78001870 con 0x7efd7c0acd00 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:49.652 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3663438216 0 0) 0x7efd7c1d70e0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 >> v1:172.21.15.79:6790/0 conn(0x7efd7c0acd00 legacy=0x7efd7c1d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 >> v1:172.21.15.79:6789/0 conn(0x7efd7c09ac40 legacy=0x7efd7c0b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 monclient: found mon.b 2024-03-29T10:47:49.653 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c1d7d10 con 0x7efd7c0af4c0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 5 monclient: authenticate success, global_id 4102 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 10 monclient: _renew_subs 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c139730 con 0x7efd7c0af4c0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd74002050 con 0x7efd7c0af4c0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7efd74003b30 con 0x7efd7c0af4c0 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.654 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 monclient: dump: 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd737fe640 4 set_mon_vals no callback set 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 10 monclient: _renew_subs 2024-03-29T10:47:49.655 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efd740040a0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c1e3ac0 con 0x7efd7c0af4c0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd83a55640 1 librados: init done 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1421082584 0 0) 0x7efd74004350 con 0x7efd7c0af4c0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 10 client.4102.objecter ms_dispatch 0x7efd7c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 3 client.4102.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 3 client.4102.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.652+0000 7efd71ffb640 20 client.4102.objecter dump_active .. 0 homeless 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.653+0000 7efd83a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.653+0000 7efd83a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.653+0000 7efd83a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.653+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c1d7270 con 0x7efd7c0af4c0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.655+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efd74004510 con 0x7efd7c0af4c0 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.655+0000 7efd71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.656 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.655+0000 7efd71ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:49.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.725+0000 7efd83a55640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e014f81f-8d58-44ef-8d6e-2c0895a7c3bd", "id": 0}] 2024-03-29T10:47:49.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.725+0000 7efd83a55640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e014f81f-8d58-44ef-8d6e-2c0895a7c3bd", "id": 0}] 2024-03-29T10:47:49.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.725+0000 7efd83a55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.707 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.725+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e014f81f-8d58-44ef-8d6e-2c0895a7c3bd", "id": 0} v 0) v1 -- 0x7efd7c09b020 con 0x7efd7c0af4c0 2024-03-29T10:47:49.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd71ffb640 1 -- 172.21.15.82:0/2226433843 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e014f81f-8d58-44ef-8d6e-2c0895a7c3bd", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3076589888 0 732707515) 0x7efd74006450 con 0x7efd7c0af4c0 2024-03-29T10:47:49.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e014f81f-8d58-44ef-8d6e-2c0895a7c3bd", "id": 0}] 2024-03-29T10:47:49.710 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd71ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:49.710 INFO:teuthology.orchestra.run.smithi082.stdout:0 2024-03-29T10:47:49.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 10 librados: watch_flush enter 2024-03-29T10:47:49.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 10 librados: watch_flush exit 2024-03-29T10:47:49.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 20 client.4102.objecter shutdown clearing up homeless session... 2024-03-29T10:47:49.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 10 client.4102.objecter successfully canceled tick 2024-03-29T10:47:49.711 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 10 monclient: shutdown 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.728+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 >> v1:172.21.15.82:6789/0 conn(0x7efd7c0af4c0 legacy=0x7efd7c139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.729+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 shutdown_connections 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.729+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 >> 172.21.15.82:0/2226433843 conn(0x7efd7c066b00 msgr2=0x7efd7c13c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.729+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 shutdown_connections 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.729+0000 7efd83a55640 1 -- 172.21.15.82:0/2226433843 wait complete. 2024-03-29T10:47:49.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.729+0000 7efd83a55640 1 librados: shutdown 2024-03-29T10:47:49.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 3dcd6634-e6ea-4862-aef6-f70ff756ca74 1 2024-03-29T10:47:49.830 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.838+0000 7f98c7388640 10 monclient: get_monmap_and_config 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.838+0000 7f98c7388640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.838+0000 7f98c7388640 10 monclient: monmap: 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:49.839287+0000 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:49.839287+0000 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.838+0000 7f98c7388640 1 Processor -- start 2024-03-29T10:47:49.831 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 1 -- start start 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient: init 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient(hunting): picked mon.noname-c con 0x7f98c009ac40 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient(hunting): picked mon.noname-a con 0x7f98c009c870 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient(hunting): picked mon.noname-b con 0x7f98c00b4360 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c009dec0 con 0x7f98c009c870 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c009da30 con 0x7f98c009ac40 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c00b39c0 con 0x7f98c00b4360 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c7388640 10 monclient(hunting): authenticate will time out at 832.011009s 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c5124640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f98c009ac40 0x7f98c00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57364/0 (socket says 172.21.15.82:57364) 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.839+0000 7f98c5124640 1 -- 172.21.15.82:0/798398395 learned_addr learned my addr 172.21.15.82:0/798398395 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:49.832 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316783903 0 0) 0x7f98c009da30 con 0x7f98c009ac40 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac003520 con 0x7f98c009ac40 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550755734 0 0) 0x7f98bc001110 con 0x7f98c009c870 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c009da30 con 0x7f98c009c870 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353701613 0 0) 0x7f98c00b39c0 con 0x7f98c00b4360 2024-03-29T10:47:49.833 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc001110 con 0x7f98c00b4360 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281826132 0 0) 0x7f98ac003520 con 0x7f98c009ac40 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 >> v1:172.21.15.79:6790/0 conn(0x7f98c00b4360 legacy=0x7f98c014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 >> v1:172.21.15.79:6789/0 conn(0x7f98c009c870 legacy=0x7f98c013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c00b5160 con 0x7f98c009ac40 2024-03-29T10:47:49.834 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f98b0001690 con 0x7f98c009ac40 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: dump: 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.835 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98c7388640 5 monclient: authenticate success, global_id 4105 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98c7388640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98b0001870 con 0x7f98c009ac40 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 1 -- 172.21.15.82:0/798398395 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f98b0001c80 con 0x7f98c009ac40 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98c7388640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98c7388640 4 set_mon_vals no callback set 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.840+0000 7f98b7fff640 10 monclient: dump: 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.836 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 10 monclient: shutdown 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 1 -- 172.21.15.82:0/798398395 >> v1:172.21.15.82:6789/0 conn(0x7f98c009ac40 legacy=0x7f98c00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 1 -- 172.21.15.82:0/798398395 shutdown_connections 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 1 -- 172.21.15.82:0/798398395 >> 172.21.15.82:0/798398395 conn(0x7f98c0066b00 msgr2=0x7f98c0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 1 -- 172.21.15.82:0/798398395 shutdown_connections 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.841+0000 7f98c7388640 1 -- 172.21.15.82:0/798398395 wait complete. 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 10 monclient: build_initial_monmap 2024-03-29T10:47:49.837 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 10 monclient: monmap: 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 1 librados: starting msgr at 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 1 librados: starting objecter 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 1 Processor -- start 2024-03-29T10:47:49.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 1 -- start start 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 1 librados: setting wanted keys 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 1 librados: calling monclient init 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 10 monclient: init 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.842+0000 7f98c7388640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient(hunting): picked mon.noname-c con 0x7f98c009ac40 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient(hunting): picked mon.noname-a con 0x7f98c01ccb70 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient(hunting): picked mon.noname-b con 0x7f98c01425f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c01cd520 con 0x7f98c01ccb70 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c00acd00 con 0x7f98c01425f0 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c01cd000 con 0x7f98c009ac40 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient(hunting): authenticate will time out at 832.014304s 2024-03-29T10:47:49.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c4923640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f98c01ccb70 0x7f98c013adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.82:41750/0 (socket says 172.21.15.82:41750) 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c5124640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f98c009ac40 0x7f98c01da690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.82:55556/0 (socket says 172.21.15.82:55556) 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c5925640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f98c01425f0 0x7f98c01e29d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57376/0 (socket says 172.21.15.82:57376) 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c4923640 1 -- 172.21.15.82:0/4161649760 learned_addr learned my addr 172.21.15.82:0/4161649760 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 client.?.objecter ms_handle_connect 0x7f98c01ccb70 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 client.?.objecter ms_handle_connect 0x7f98c01425f0 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 client.?.objecter ms_handle_connect 0x7f98c009ac40 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796609480 0 0) 0x7f98c01cd520 con 0x7f98c01ccb70 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:47:49.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a4004eb0 con 0x7f98c01ccb70 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762892986 0 0) 0x7f98c00acd00 con 0x7f98c01425f0 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c01cd520 con 0x7f98c01425f0 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347575012 0 0) 0x7f98c01cd000 con 0x7f98c009ac40 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c00acd00 con 0x7f98c009ac40 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051457800 0 0) 0x7f98a4004eb0 con 0x7f98c01ccb70 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c01cd000 con 0x7f98c01ccb70 2024-03-29T10:47:49.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f98bc001a20 con 0x7f98c01ccb70 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:49.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 691666336 0 0) 0x7f98c01cd520 con 0x7f98c01425f0 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98a4004eb0 con 0x7f98c01425f0 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f98b8004110 con 0x7f98c01425f0 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.843 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4049844962 0 0) 0x7f98c00acd00 con 0x7f98c009ac40 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c01cd520 con 0x7f98c009ac40 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f98b00019e0 con 0x7f98c009ac40 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.844 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3339535429 0 0) 0x7f98c01cd000 con 0x7f98c01ccb70 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 >> v1:172.21.15.79:6790/0 conn(0x7f98c009ac40 legacy=0x7f98c01da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 >> v1:172.21.15.82:6789/0 conn(0x7f98c01425f0 legacy=0x7f98c01e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 monclient: found mon.a 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c00b1af0 con 0x7f98c01ccb70 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:49.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 5 monclient: authenticate success, global_id 4125 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient: _renew_subs 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.843+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c01cc300 con 0x7f98c01ccb70 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98bc001c80 con 0x7f98c01ccb70 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f98bc003a60 con 0x7f98c01ccb70 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 10 monclient: dump: 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:49.846 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f98bc004010 con 0x7f98c01ccb70 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b77fe640 4 set_mon_vals no callback set 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98c7388640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98c7388640 10 monclient: _renew_subs 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98c7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c01daf80 con 0x7f98c01ccb70 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98c7388640 1 librados: init done 2024-03-29T10:47:49.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2055687118 0 0) 0x7f98bc005b70 con 0x7f98c01ccb70 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 10 client.4125.objecter ms_dispatch 0x7f98c0000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.844+0000 7f98b5ffb640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.845+0000 7f98c7388640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.845+0000 7f98c7388640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.845+0000 7f98c7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.845+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c01cd190 con 0x7f98c01ccb70 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.847+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f98bc0063a0 con 0x7f98c01ccb70 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.847+0000 7f98b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:49.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.847+0000 7f98b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:49.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.917+0000 7f98c7388640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3dcd6634-e6ea-4862-aef6-f70ff756ca74", "id": 1}] 2024-03-29T10:47:49.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.918+0000 7f98c7388640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3dcd6634-e6ea-4862-aef6-f70ff756ca74", "id": 1}] 2024-03-29T10:47:49.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.918+0000 7f98c7388640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:49.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.918+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3dcd6634-e6ea-4862-aef6-f70ff756ca74", "id": 1} v 0) v1 -- 0x7f98c009b020 con 0x7f98c01ccb70 2024-03-29T10:47:49.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.919+0000 7f98b5ffb640 1 -- 172.21.15.82:0/4161649760 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3dcd6634-e6ea-4862-aef6-f70ff756ca74", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3245135750 0 940483276) 0x7f98bc0065c0 con 0x7f98c01ccb70 2024-03-29T10:47:49.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.919+0000 7f98b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3dcd6634-e6ea-4862-aef6-f70ff756ca74", "id": 1}] 2024-03-29T10:47:49.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.919+0000 7f98b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:49.901 INFO:teuthology.orchestra.run.smithi082.stdout:1 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 10 librados: watch_flush enter 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 10 librados: watch_flush exit 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 10 client.4125.objecter successfully canceled tick 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 10 monclient: shutdown 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 >> v1:172.21.15.79:6789/0 conn(0x7f98c01ccb70 legacy=0x7f98c013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 shutdown_connections 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 >> 172.21.15.82:0/4161649760 conn(0x7f98c0066b00 msgr2=0x7f98c0066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 shutdown_connections 2024-03-29T10:47:49.903 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 1 -- 172.21.15.82:0/4161649760 wait complete. 2024-03-29T10:47:49.904 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:49.920+0000 7f98c7388640 1 librados: shutdown 2024-03-29T10:47:49.912 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new f3cdacad-8901-4794-ad1c-1368e5a42e0a 2 2024-03-29T10:47:49.995 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.013+0000 7feb80432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.029+0000 7fb508be6640 10 monclient: get_monmap_and_config 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.029+0000 7fb508be6640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.029+0000 7fb508be6640 10 monclient: monmap: 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:50.030511+0000 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:50.030511+0000 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:50.012 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:50.013 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 1 Processor -- start 2024-03-29T10:47:50.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 1 -- start start 2024-03-29T10:47:50.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient: init 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient(hunting): picked mon.noname-a con 0x7fb50409ac40 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient(hunting): picked mon.noname-b con 0x7fb50409c870 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5040b4360 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb50409dec0 con 0x7fb50409ac40 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb50409da30 con 0x7fb5040b4360 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5040b39c0 con 0x7fb50409c870 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.030+0000 7fb508be6640 10 monclient(hunting): authenticate will time out at 832.202216s 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb502d9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb5040b4360 0x7fb50414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57380/0 (socket says 172.21.15.82:57380) 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb502d9d640 1 -- 172.21.15.82:0/1186777403 learned_addr learned my addr 172.21.15.82:0/1186777403 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405708806 0 0) 0x7fb50409da30 con 0x7fb5040b4360 2024-03-29T10:47:50.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec003520 con 0x7fb5040b4360 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762192070 0 0) 0x7fb4f4001110 con 0x7fb50409ac40 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb50409da30 con 0x7fb50409ac40 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511967353 0 0) 0x7fb5040b39c0 con 0x7fb50409c870 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f4001110 con 0x7fb50409c870 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3507422152 0 0) 0x7fb4ec003520 con 0x7fb5040b4360 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 >> v1:172.21.15.79:6790/0 conn(0x7fb50409c870 legacy=0x7fb50413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 >> v1:172.21.15.79:6789/0 conn(0x7fb50409ac40 legacy=0x7fb5040b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 monclient: found mon.noname-c 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5040b5160 con 0x7fb5040b4360 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb508be6640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb4fc001a10 con 0x7fb5040b4360 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: dump: 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.026 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb508be6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4fc001bf0 con 0x7fb5040b4360 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb508be6640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 1 -- 172.21.15.82:0/1186777403 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb4fc002040 con 0x7fb5040b4360 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb508be6640 4 set_mon_vals no callback set 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.031+0000 7fb50159a640 10 monclient: dump: 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.027 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 10 monclient: shutdown 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 1 -- 172.21.15.82:0/1186777403 >> v1:172.21.15.82:6789/0 conn(0x7fb5040b4360 legacy=0x7fb50414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 1 -- 172.21.15.82:0/1186777403 shutdown_connections 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 1 -- 172.21.15.82:0/1186777403 >> 172.21.15.82:0/1186777403 conn(0x7fb504066b00 msgr2=0x7fb504066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 1 -- 172.21.15.82:0/1186777403 shutdown_connections 2024-03-29T10:47:50.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.032+0000 7fb508be6640 1 -- 172.21.15.82:0/1186777403 wait complete. 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 10 monclient: monmap: 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.029 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 1 librados: starting msgr at 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 1 librados: starting objecter 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 1 Processor -- start 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 1 -- start start 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 1 librados: setting wanted keys 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 1 librados: calling monclient init 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.033+0000 7fb508be6640 10 monclient: init 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient(hunting): picked mon.noname-a con 0x7fb50409ac40 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.030 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient(hunting): picked mon.noname-c con 0x7fb504140cc0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient(hunting): picked mon.noname-b con 0x7fb504138e10 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5041d7330 con 0x7fb50409ac40 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb504142b60 con 0x7fb504138e10 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb504142dd0 con 0x7fb504140cc0 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient(hunting): authenticate will time out at 832.205326s 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb502d9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb504138e10 0x7fb504139270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57392/0 (socket says 172.21.15.82:57392) 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb502d9d640 1 -- 172.21.15.82:0/455254674 learned_addr learned my addr 172.21.15.82:0/455254674 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb504138e10 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb50409ac40 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb504140cc0 2024-03-29T10:47:50.031 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427495641 0 0) 0x7fb504142b60 con 0x7fb504138e10 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e4004eb0 con 0x7fb504138e10 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991465337 0 0) 0x7fb5041d7330 con 0x7fb50409ac40 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb504142b60 con 0x7fb50409ac40 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515646256 0 0) 0x7fb504142dd0 con 0x7fb504140cc0 2024-03-29T10:47:50.032 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5041d7330 con 0x7fb504140cc0 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295913211 0 0) 0x7fb4e4004eb0 con 0x7fb504138e10 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb504142dd0 con 0x7fb504138e10 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb4fc001f40 con 0x7fb504138e10 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.033 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2593580 0 0) 0x7fb504142b60 con 0x7fb50409ac40 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e4004eb0 con 0x7fb50409ac40 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450276166 0 0) 0x7fb5041d7330 con 0x7fb504140cc0 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb504142b60 con 0x7fb504140cc0 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb4f4001a40 con 0x7fb50409ac40 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:50.034 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb4f8001770 con 0x7fb504140cc0 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.035 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1927017806 0 0) 0x7fb504142dd0 con 0x7fb504138e10 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 >> v1:172.21.15.79:6790/0 conn(0x7fb504140cc0 legacy=0x7fb5041d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 >> v1:172.21.15.79:6789/0 conn(0x7fb50409ac40 legacy=0x7fb5041418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 monclient: found mon.b 2024-03-29T10:47:50.036 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5040afcd0 con 0x7fb504138e10 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb4f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient: _renew_subs 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5041d7ed0 con 0x7fb504138e10 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient: _renew_subs 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5041e3c80 con 0x7fb504138e10 2024-03-29T10:47:50.037 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.034+0000 7fb508be6640 1 librados: init done 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4fc004e10 con 0x7fb504138e10 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb4fc005120 con 0x7fb504138e10 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb500d99640 4 set_mon_vals no callback set 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 10 monclient: dump: 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.038 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb4fc005610 con 0x7fb504138e10 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2846518097 0 0) 0x7fb4fc005ce0 con 0x7fb504138e10 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 10 client.4114.objecter ms_dispatch 0x7fb504000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 3 client.4114.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 3 client.4114.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.035+0000 7fb4f2ffd640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.036+0000 7fb508be6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.036+0000 7fb508be6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.036+0000 7fb508be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.036+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e8005080 con 0x7fb504138e10 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.039+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb4fc005fc0 con 0x7fb504138e10 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.039+0000 7fb4f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.039 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.039+0000 7fb4f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.040 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.045+0000 7f793a01e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:47:50.049 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.067+0000 7feb80432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:47:50.081 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.099+0000 7f793a01e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:47:50.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.110+0000 7fb508be6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f3cdacad-8901-4794-ad1c-1368e5a42e0a", "id": 2}] 2024-03-29T10:47:50.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.110+0000 7fb508be6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f3cdacad-8901-4794-ad1c-1368e5a42e0a", "id": 2}] 2024-03-29T10:47:50.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.110+0000 7fb508be6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.110+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f3cdacad-8901-4794-ad1c-1368e5a42e0a", "id": 2} v 0) v1 -- 0x7fb4e80058b0 con 0x7fb504138e10 2024-03-29T10:47:50.093 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.111+0000 7feb80432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:47:50.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.114+0000 7fb4f2ffd640 1 -- 172.21.15.82:0/455254674 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3cdacad-8901-4794-ad1c-1368e5a42e0a", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 166334289 0 216599125) 0x7fb4fc0058c0 con 0x7fb504138e10 2024-03-29T10:47:50.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.114+0000 7fb4f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f3cdacad-8901-4794-ad1c-1368e5a42e0a", "id": 2}] 2024-03-29T10:47:50.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.114+0000 7fb4f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.096 INFO:teuthology.orchestra.run.smithi082.stdout:2 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 10 librados: watch_flush enter 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 10 librados: watch_flush exit 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 10 client.4114.objecter successfully canceled tick 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 10 monclient: shutdown 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 >> v1:172.21.15.82:6789/0 conn(0x7fb504138e10 legacy=0x7fb504139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 shutdown_connections 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 >> 172.21.15.82:0/455254674 conn(0x7fb504066b00 msgr2=0x7fb5040b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 shutdown_connections 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 1 -- 172.21.15.82:0/455254674 wait complete. 2024-03-29T10:47:50.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.115+0000 7fb508be6640 1 librados: shutdown 2024-03-29T10:47:50.107 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new cfba379b-8173-47fb-b761-87462a52e762 3 2024-03-29T10:47:50.129 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.147+0000 7f793a01e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:47:50.147 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.165+0000 7feb80432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:47:50.185 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.204+0000 7f793a01e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:47:50.190 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.208+0000 7feb80432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.224+0000 7f41f393b640 10 monclient: get_monmap_and_config 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.224+0000 7f41f393b640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.224+0000 7f41f393b640 10 monclient: monmap: 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:50.225046+0000 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:50.225046+0000 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:50.206 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:50.207 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.224+0000 7f41f393b640 1 Processor -- start 2024-03-29T10:47:50.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.224+0000 7f41f393b640 1 -- start start 2024-03-29T10:47:50.216 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.224+0000 7f41f393b640 10 monclient: init 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec09c870 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec0b4360 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec09dec0 con 0x7f41ec09ac40 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec09da30 con 0x7f41ec09c870 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0b39c0 con 0x7f41ec0b4360 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f393b640 10 monclient(hunting): authenticate will time out at 832.396801s 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f0ed6640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f41ec09c870 0x7f41ec13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57406/0 (socket says 172.21.15.82:57406) 2024-03-29T10:47:50.217 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41f0ed6640 1 -- 172.21.15.82:0/3625739602 learned_addr learned my addr 172.21.15.82:0/3625739602 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.225+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387653434 0 0) 0x7f41ec09da30 con 0x7f41ec09c870 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d8003520 con 0x7f41ec09c870 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145116129 0 0) 0x7f41ec0b39c0 con 0x7f41ec0b4360 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:47:50.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec09da30 con 0x7f41ec0b4360 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065629422 0 0) 0x7f41dc001110 con 0x7f41ec09ac40 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec0b39c0 con 0x7f41ec09ac40 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1605085694 0 0) 0x7f41d8003520 con 0x7f41ec09c870 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 >> v1:172.21.15.79:6790/0 conn(0x7f41ec0b4360 legacy=0x7f41ec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 >> v1:172.21.15.79:6789/0 conn(0x7f41ec09ac40 legacy=0x7f41ec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0b5160 con 0x7f41ec09c870 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f41e8001690 con 0x7f41ec09c870 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: dump: 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.220 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e8001860 con 0x7f41ec09c870 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 1 -- 172.21.15.82:0/3625739602 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f41e8001c30 con 0x7f41ec09c870 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 4 set_mon_vals no callback set 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41e3fff640 10 monclient: dump: 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.221 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 10 monclient: shutdown 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.226+0000 7f41f393b640 1 -- 172.21.15.82:0/3625739602 >> v1:172.21.15.82:6789/0 conn(0x7f41ec09c870 legacy=0x7f41ec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 1 -- 172.21.15.82:0/3625739602 shutdown_connections 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 1 -- 172.21.15.82:0/3625739602 >> 172.21.15.82:0/3625739602 conn(0x7f41ec066b00 msgr2=0x7f41ec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 1 -- 172.21.15.82:0/3625739602 shutdown_connections 2024-03-29T10:47:50.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 1 -- 172.21.15.82:0/3625739602 wait complete. 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 10 monclient: monmap: 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 1 librados: starting msgr at 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.227+0000 7f41f393b640 1 librados: starting objecter 2024-03-29T10:47:50.223 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 Processor -- start 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 -- start start 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 librados: setting wanted keys 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 librados: calling monclient init 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient: init 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec09ac40 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec13ffa0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec0b0490 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0add50 con 0x7f41ec13ffa0 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0b1720 con 0x7f41ec09ac40 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec1e4c00 con 0x7f41ec0b0490 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f393b640 10 monclient(hunting): authenticate will time out at 832.399898s 2024-03-29T10:47:50.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f16d7640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f41ec09ac40 0x7f41ec0b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57418/0 (socket says 172.21.15.82:57418) 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41f16d7640 1 -- 172.21.15.82:0/1831466389 learned_addr learned my addr 172.21.15.82:0/1831466389 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 client.?.objecter ms_handle_connect 0x7f41ec09ac40 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 client.?.objecter ms_handle_connect 0x7f41ec13ffa0 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 client.?.objecter ms_handle_connect 0x7f41ec0b0490 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491961589 0 0) 0x7f41ec0b1720 con 0x7f41ec09ac40 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.228+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d0004eb0 con 0x7f41ec09ac40 2024-03-29T10:47:50.225 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107251898 0 0) 0x7f41ec0add50 con 0x7f41ec13ffa0 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec0b1720 con 0x7f41ec13ffa0 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163546429 0 0) 0x7f41ec1e4c00 con 0x7f41ec0b0490 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec0add50 con 0x7f41ec0b0490 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 810084848 0 0) 0x7f41d0004eb0 con 0x7f41ec09ac40 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ec1e4c00 con 0x7f41ec09ac40 2024-03-29T10:47:50.226 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f41dc001980 con 0x7f41ec09ac40 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.227 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2242282794 0 0) 0x7f41ec0b1720 con 0x7f41ec13ffa0 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41d0004eb0 con 0x7f41ec13ffa0 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f41e8001b70 con 0x7f41ec13ffa0 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.228 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4012897215 0 0) 0x7f41ec0add50 con 0x7f41ec0b0490 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ec0b1720 con 0x7f41ec0b0490 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f41e4001b50 con 0x7f41ec0b0490 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.229 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 562653524 0 0) 0x7f41ec1e4c00 con 0x7f41ec09ac40 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 >> v1:172.21.15.79:6790/0 conn(0x7f41ec0b0490 legacy=0x7f41ec0ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 >> v1:172.21.15.79:6789/0 conn(0x7f41ec13ffa0 legacy=0x7f41ec0ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 monclient: found mon.b 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec1e5da0 con 0x7f41ec09ac40 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.230 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 10 monclient: _renew_subs 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec1e4d90 con 0x7f41ec09ac40 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41dc0035f0 con 0x7f41ec09ac40 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f41dc003a00 con 0x7f41ec09ac40 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 monclient: dump: 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.231 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41dc003f70 con 0x7f41ec09ac40 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e37fe640 4 set_mon_vals no callback set 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 10 monclient: _renew_subs 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec1e52c0 con 0x7f41ec09ac40 2024-03-29T10:47:50.232 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41f393b640 1 librados: init done 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4152283236 0 0) 0x7f41dc005c10 con 0x7f41ec09ac40 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 10 client.4120.objecter ms_dispatch 0x7f41ec000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 3 client.4120.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 3 client.4120.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.229+0000 7f41e1ffb640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.230+0000 7f41f393b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.230+0000 7f41f393b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.230+0000 7f41f393b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.230+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec09b020 con 0x7f41ec09ac40 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.232+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41dc005780 con 0x7f41ec09ac40 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.232+0000 7f41e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.232+0000 7f41e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.234 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.249+0000 7f793a01e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:47:50.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.304+0000 7f41f393b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cfba379b-8173-47fb-b761-87462a52e762", "id": 3}] 2024-03-29T10:47:50.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.304+0000 7f41f393b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cfba379b-8173-47fb-b761-87462a52e762", "id": 3}] 2024-03-29T10:47:50.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.304+0000 7f41f393b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.304+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cfba379b-8173-47fb-b761-87462a52e762", "id": 3} v 0) v1 -- 0x7f41ec04f730 con 0x7f41ec09ac40 2024-03-29T10:47:50.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41e1ffb640 1 -- 172.21.15.82:0/1831466389 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cfba379b-8173-47fb-b761-87462a52e762", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3470444493 0 525046306) 0x7f41dc005a20 con 0x7f41ec09ac40 2024-03-29T10:47:50.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cfba379b-8173-47fb-b761-87462a52e762", "id": 3}] 2024-03-29T10:47:50.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.290 INFO:teuthology.orchestra.run.smithi082.stdout:3 2024-03-29T10:47:50.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41f393b640 10 librados: watch_flush enter 2024-03-29T10:47:50.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41f393b640 10 librados: watch_flush exit 2024-03-29T10:47:50.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41f393b640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.308+0000 7f41f393b640 10 client.4120.objecter successfully canceled tick 2024-03-29T10:47:50.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 10 monclient: shutdown 2024-03-29T10:47:50.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.292 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 >> v1:172.21.15.82:6789/0 conn(0x7f41ec09ac40 legacy=0x7f41ec0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.292 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 shutdown_connections 2024-03-29T10:47:50.292 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 >> 172.21.15.82:0/1831466389 conn(0x7f41ec066b00 msgr2=0x7f41ec13e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.292 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 shutdown_connections 2024-03-29T10:47:50.292 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 1 -- 172.21.15.82:0/1831466389 wait complete. 2024-03-29T10:47:50.292 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.309+0000 7f41f393b640 1 librados: shutdown 2024-03-29T10:47:50.301 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 729faf03-f69a-40f8-9ae9-1b98771fa643 4 2024-03-29T10:47:50.305 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.323+0000 7feb80432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:47:50.350 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.368+0000 7f793a01e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:47:50.374 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.392+0000 7feb80432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:50.419 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.437+0000 7feb80432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:47:50.423 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.441+0000 7f793a01e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:50.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.457+0000 7f6adb115640 10 monclient: get_monmap_and_config 2024-03-29T10:47:50.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.457+0000 7f6adb115640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.457+0000 7f6adb115640 10 monclient: monmap: 2024-03-29T10:47:50.439 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.439 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:50.458410+0000 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:50.458410+0000 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:50.440 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 1 Processor -- start 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 1 -- start start 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient: init 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad409ac40 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad409c870 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad40b4360 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad409dec0 con 0x7f6ad409c870 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad409da30 con 0x7f6ad40b4360 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad40b39c0 con 0x7f6ad409ac40 2024-03-29T10:47:50.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.458+0000 7f6adb115640 10 monclient(hunting): authenticate will time out at 832.630150s 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad96b2640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6ad40b4360 0x7f6ad414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57426/0 (socket says 172.21.15.82:57426) 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad96b2640 1 -- 172.21.15.82:0/1957837670 learned_addr learned my addr 172.21.15.82:0/1957837670 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366431602 0 0) 0x7f6ad409da30 con 0x7f6ad40b4360 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac4003520 con 0x7f6ad40b4360 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147193566 0 0) 0x7f6ac0001110 con 0x7f6ad409c870 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:47:50.451 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad409da30 con 0x7f6ad409c870 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194529174 0 0) 0x7f6ad40b39c0 con 0x7f6ad409ac40 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac0001110 con 0x7f6ad409ac40 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249317061 0 0) 0x7f6ad409da30 con 0x7f6ad409c870 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 >> v1:172.21.15.79:6790/0 conn(0x7f6ad409ac40 legacy=0x7f6ad40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 >> v1:172.21.15.82:6789/0 conn(0x7f6ad40b4360 legacy=0x7f6ad414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 monclient: found mon.noname-a 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad40b5160 con 0x7f6ad409c870 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.452 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6adb115640 5 monclient: authenticate success, global_id 4140 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6ac0001660 con 0x7f6ad409c870 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6adb115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: dump: 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.453 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6adb115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ac0001840 con 0x7f6ad409c870 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 1 -- 172.21.15.82:0/1957837670 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6ac0001c90 con 0x7f6ad409c870 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6adb115640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6adb115640 4 set_mon_vals no callback set 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.459+0000 7f6ad37fe640 10 monclient: dump: 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.454 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 10 monclient: shutdown 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 1 -- 172.21.15.82:0/1957837670 >> v1:172.21.15.79:6789/0 conn(0x7f6ad409c870 legacy=0x7f6ad413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 1 -- 172.21.15.82:0/1957837670 shutdown_connections 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 1 -- 172.21.15.82:0/1957837670 >> 172.21.15.82:0/1957837670 conn(0x7f6ad4066b00 msgr2=0x7f6ad4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 1 -- 172.21.15.82:0/1957837670 shutdown_connections 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 1 -- 172.21.15.82:0/1957837670 wait complete. 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.460+0000 7f6adb115640 10 monclient: monmap: 2024-03-29T10:47:50.455 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 1 librados: starting msgr at 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 1 librados: starting objecter 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 1 Processor -- start 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 1 -- start start 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 1 librados: setting wanted keys 2024-03-29T10:47:50.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 1 librados: calling monclient init 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.461+0000 7f6adb115640 10 monclient: init 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient(hunting): picked mon.noname-b con 0x7f6ad409ac40 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient(hunting): picked mon.noname-c con 0x7f6ad4139b60 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient(hunting): picked mon.noname-a con 0x7f6ad41461a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad41402f0 con 0x7f6ad41461a0 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad41d7630 con 0x7f6ad409ac40 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad413ffa0 con 0x7f6ad4139b60 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient(hunting): authenticate will time out at 832.633350s 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad8eb1640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6ad409ac40 0x7f6ad4145a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:57440/0 (socket says 172.21.15.82:57440) 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad8eb1640 1 -- 172.21.15.82:0/240181681 learned_addr learned my addr 172.21.15.82:0/240181681 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 client.?.objecter ms_handle_connect 0x7f6ad409ac40 2024-03-29T10:47:50.457 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 client.?.objecter ms_handle_connect 0x7f6ad41461a0 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 client.?.objecter ms_handle_connect 0x7f6ad4139b60 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469719694 0 0) 0x7f6ad41d7630 con 0x7f6ad409ac40 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab8004eb0 con 0x7f6ad409ac40 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212125810 0 0) 0x7f6ad41402f0 con 0x7f6ad41461a0 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.458 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad41d7630 con 0x7f6ad41461a0 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281013095 0 0) 0x7f6ad413ffa0 con 0x7f6ad4139b60 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad41402f0 con 0x7f6ad4139b60 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488246869 0 0) 0x7f6ad41d7630 con 0x7f6ad41461a0 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ad413ffa0 con 0x7f6ad41461a0 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6acc001ab0 con 0x7f6ad41461a0 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): dump: 2024-03-29T10:47:50.459 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.460 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.461 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.461 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.461 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:50.461 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398057826 0 0) 0x7f6ab8004eb0 con 0x7f6ad409ac40 2024-03-29T10:47:50.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ad41d7630 con 0x7f6ad409ac40 2024-03-29T10:47:50.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777208443 0 0) 0x7f6ad41402f0 con 0x7f6ad4139b60 2024-03-29T10:47:50.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ab8004eb0 con 0x7f6ad4139b60 2024-03-29T10:47:50.462 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6ac8001a00 con 0x7f6ad409ac40 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): dump: 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.463 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.464 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.464 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.464 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.464 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.465 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.465 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6ac0001b50 con 0x7f6ad4139b60 2024-03-29T10:47:50.465 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient(hunting): dump: 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.466 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2013766106 0 0) 0x7f6ad413ffa0 con 0x7f6ad41461a0 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 >> v1:172.21.15.79:6790/0 conn(0x7f6ad4139b60 legacy=0x7f6ad40af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 >> v1:172.21.15.82:6789/0 conn(0x7f6ad409ac40 legacy=0x7f6ad4145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 monclient: found mon.a 2024-03-29T10:47:50.467 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad40b0070 con 0x7f6ad41461a0 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6ad17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 5 monclient: authenticate success, global_id 4143 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient: _renew_subs 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.468 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.462+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad40b0320 con 0x7f6ad41461a0 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6adb115640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6adb115640 10 monclient: _renew_subs 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6adb115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad41d7080 con 0x7f6ad41461a0 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6adb115640 1 librados: init done 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6acc002020 con 0x7f6ad41461a0 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6acc004d00 con 0x7f6ad41461a0 2024-03-29T10:47:50.469 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad2ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 10 monclient: dump: 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.470 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6acc0051f0 con 0x7f6ad41461a0 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3789209636 0 0) 0x7f6acc006990 con 0x7f6ad41461a0 2024-03-29T10:47:50.471 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 10 client.4143.objecter ms_dispatch 0x7f6ad4000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.463+0000 7f6ad17fa640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.464+0000 7f6adb115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.464+0000 7f6adb115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.464+0000 7f6adb115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.464+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab4005080 con 0x7f6ad41461a0 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.466+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6acc006530 con 0x7f6ad41461a0 2024-03-29T10:47:50.472 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.466+0000 7f6ad17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.473 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.466+0000 7f6ad17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.473 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.489+0000 7f793a01e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:47:50.474 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.490+0000 7feb80432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:47:50.518 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.536+0000 7f6adb115640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "729faf03-f69a-40f8-9ae9-1b98771fa643", "id": 4}] 2024-03-29T10:47:50.518 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.536+0000 7f6adb115640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "729faf03-f69a-40f8-9ae9-1b98771fa643", "id": 4}] 2024-03-29T10:47:50.518 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.536+0000 7f6adb115640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.518 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.536+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "729faf03-f69a-40f8-9ae9-1b98771fa643", "id": 4} v 0) v1 -- 0x7f6ab40058b0 con 0x7f6ad41461a0 2024-03-29T10:47:50.519 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.536+0000 7feb80432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:47:50.520 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.538+0000 7f6ad17fa640 1 -- 172.21.15.82:0/240181681 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "729faf03-f69a-40f8-9ae9-1b98771fa643", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 99157532 0 1697012583) 0x7f6acc005680 con 0x7f6ad41461a0 2024-03-29T10:47:50.520 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.538+0000 7f6ad17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "729faf03-f69a-40f8-9ae9-1b98771fa643", "id": 4}] 2024-03-29T10:47:50.520 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.538+0000 7f6ad17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.520 INFO:teuthology.orchestra.run.smithi082.stdout:4 2024-03-29T10:47:50.521 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.538+0000 7f6adb115640 10 librados: watch_flush enter 2024-03-29T10:47:50.521 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.538+0000 7f6adb115640 10 librados: watch_flush exit 2024-03-29T10:47:50.521 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.538+0000 7f6adb115640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.521 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 10 client.4143.objecter successfully canceled tick 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 10 monclient: shutdown 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 >> v1:172.21.15.79:6789/0 conn(0x7f6ad41461a0 legacy=0x7f6ad41e24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 shutdown_connections 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 >> 172.21.15.82:0/240181681 conn(0x7f6ad4066b00 msgr2=0x7f6ad40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 shutdown_connections 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 1 -- 172.21.15.82:0/240181681 wait complete. 2024-03-29T10:47:50.522 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.539+0000 7f6adb115640 1 librados: shutdown 2024-03-29T10:47:50.525 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.543+0000 7f793a01e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:47:50.531 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 18c4cbac-2e00-43dd-ba9d-15754f0a644a 5 2024-03-29T10:47:50.574 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.592+0000 7f793a01e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:47:50.679 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.688+0000 7fe5632a8640 10 monclient: get_monmap_and_config 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.688+0000 7fe5632a8640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.688+0000 7fe5632a8640 10 monclient: monmap: 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:50.689069+0000 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:50.689069+0000 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:50.680 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.688+0000 7fe5632a8640 1 Processor -- start 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.688+0000 7fe5632a8640 1 -- start start 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.688+0000 7fe5632a8640 10 monclient: init 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe55c09ac40 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe55c09c870 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe55c0b4360 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c09dec0 con 0x7fe55c0b4360 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c09da30 con 0x7fe55c09ac40 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c0b39c0 con 0x7fe55c09c870 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe5632a8640 10 monclient(hunting): authenticate will time out at 832.860810s 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe561044640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe55c09ac40 0x7fe55c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:38706/0 (socket says 172.21.15.82:38706) 2024-03-29T10:47:50.681 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe561044640 1 -- 172.21.15.82:0/1419767485 learned_addr learned my addr 172.21.15.82:0/1419767485 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418805758 0 0) 0x7fe55c09da30 con 0x7fe55c09ac40 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe553fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe553fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.689+0000 7fe553fff640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe548003520 con 0x7fe55c09ac40 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939747464 0 0) 0x7fe554001490 con 0x7fe55c0b4360 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c09da30 con 0x7fe55c0b4360 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441778695 0 0) 0x7fe55c0b39c0 con 0x7fe55c09c870 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.682 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554001490 con 0x7fe55c09c870 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1998458462 0 0) 0x7fe548003520 con 0x7fe55c09ac40 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 >> v1:172.21.15.79:6790/0 conn(0x7fe55c09c870 legacy=0x7fe55c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 >> v1:172.21.15.79:6789/0 conn(0x7fe55c0b4360 legacy=0x7fe55c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 monclient: found mon.noname-c 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c0b5160 con 0x7fe55c09ac40 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.683 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe558001690 con 0x7fe55c09ac40 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: dump: 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.684 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe558001870 con 0x7fe55c09ac40 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 1 -- 172.21.15.82:0/1419767485 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe558001cc0 con 0x7fe55c09ac40 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 4 set_mon_vals no callback set 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe553fff640 10 monclient: dump: 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.685 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 10 monclient: shutdown 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 1 -- 172.21.15.82:0/1419767485 >> v1:172.21.15.82:6789/0 conn(0x7fe55c09ac40 legacy=0x7fe55c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 1 -- 172.21.15.82:0/1419767485 shutdown_connections 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.690+0000 7fe5632a8640 1 -- 172.21.15.82:0/1419767485 >> 172.21.15.82:0/1419767485 conn(0x7fe55c066b00 msgr2=0x7fe55c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.691+0000 7fe5632a8640 1 -- 172.21.15.82:0/1419767485 shutdown_connections 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.691+0000 7fe5632a8640 1 -- 172.21.15.82:0/1419767485 wait complete. 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.691+0000 7fe5632a8640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.691+0000 7fe5632a8640 10 monclient: monmap: 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:50.686 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.691+0000 7fe5632a8640 1 librados: starting msgr at 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.691+0000 7fe5632a8640 1 librados: starting objecter 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 Processor -- start 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 -- start start 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 librados: setting wanted keys 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 librados: calling monclient init 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient: init 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.687 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient(hunting): picked mon.noname-c con 0x7fe55c09ac40 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient(hunting): picked mon.noname-a con 0x7fe55c145a10 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient(hunting): picked mon.noname-b con 0x7fe55c0b01d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c1dac90 con 0x7fe55c145a10 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c1daf30 con 0x7fe55c0b01d0 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c1464b0 con 0x7fe55c09ac40 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe5632a8640 10 monclient(hunting): authenticate will time out at 832.863827s 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe561845640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe55c0b01d0 0x7fe55c1e1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:38718/0 (socket says 172.21.15.82:38718) 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe561845640 1 -- 172.21.15.82:0/932536967 learned_addr learned my addr 172.21.15.82:0/932536967 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 client.?.objecter ms_handle_connect 0x7fe55c0b01d0 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 client.?.objecter ms_handle_connect 0x7fe55c145a10 2024-03-29T10:47:50.688 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 client.?.objecter ms_handle_connect 0x7fe55c09ac40 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265546364 0 0) 0x7fe55c1daf30 con 0x7fe55c0b01d0 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe540004eb0 con 0x7fe55c0b01d0 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933920872 0 0) 0x7fe55c1dac90 con 0x7fe55c145a10 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.689 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c1daf30 con 0x7fe55c145a10 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488309158 0 0) 0x7fe55c1464b0 con 0x7fe55c09ac40 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.692+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c1dac90 con 0x7fe55c09ac40 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2802415457 0 0) 0x7fe540004eb0 con 0x7fe55c0b01d0 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe55c1464b0 con 0x7fe55c0b01d0 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe554001dd0 con 0x7fe55c0b01d0 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:50.690 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524592223 0 0) 0x7fe55c1daf30 con 0x7fe55c145a10 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe540004eb0 con 0x7fe55c145a10 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe54c001870 con 0x7fe55c145a10 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 702884100 0 0) 0x7fe55c1dac90 con 0x7fe55c09ac40 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe55c1daf30 con 0x7fe55c09ac40 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe558001b80 con 0x7fe55c09ac40 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient(hunting): dump: 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 139056887 0 0) 0x7fe540004eb0 con 0x7fe55c145a10 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 >> v1:172.21.15.79:6790/0 conn(0x7fe55c09ac40 legacy=0x7fe55c1da580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 >> v1:172.21.15.82:6789/0 conn(0x7fe55c0b01d0 legacy=0x7fe55c1e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 monclient: found mon.a 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c0b0640 con 0x7fe55c145a10 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 5 monclient: authenticate success, global_id 4149 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 10 monclient: _renew_subs 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55c1e2f70 con 0x7fe55c145a10 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe54c001d00 con 0x7fe55c145a10 2024-03-29T10:47:50.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe54c0037b0 con 0x7fe55c145a10 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 monclient: dump: 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe54c003ca0 con 0x7fe55c145a10 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5537fe640 4 set_mon_vals no callback set 2024-03-29T10:47:50.695 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 10 monclient: _renew_subs 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55c1e32e0 con 0x7fe55c145a10 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe5632a8640 1 librados: init done 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3889798137 0 0) 0x7fe54c004870 con 0x7fe55c145a10 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 10 client.4149.objecter ms_dispatch 0x7fe55c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.693+0000 7fe551ffb640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.694+0000 7fe5632a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.694+0000 7fe5632a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.694+0000 7fe5632a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.694+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55c09b020 con 0x7fe55c145a10 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.696+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe54c0041d0 con 0x7fe55c145a10 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.696+0000 7fe551ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.696+0000 7fe551ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.749 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.767+0000 7fe5632a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "18c4cbac-2e00-43dd-ba9d-15754f0a644a", "id": 5}] 2024-03-29T10:47:50.749 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.767+0000 7fe5632a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "18c4cbac-2e00-43dd-ba9d-15754f0a644a", "id": 5}] 2024-03-29T10:47:50.749 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.767+0000 7fe5632a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.749 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.767+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "18c4cbac-2e00-43dd-ba9d-15754f0a644a", "id": 5} v 0) v1 -- 0x7fe55c13e1e0 con 0x7fe55c145a10 2024-03-29T10:47:50.750 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.768+0000 7fe551ffb640 1 -- 172.21.15.82:0/932536967 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18c4cbac-2e00-43dd-ba9d-15754f0a644a", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 326161796 0 1988412176) 0x7fe54c004370 con 0x7fe55c145a10 2024-03-29T10:47:50.750 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.768+0000 7fe551ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "18c4cbac-2e00-43dd-ba9d-15754f0a644a", "id": 5}] 2024-03-29T10:47:50.750 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.768+0000 7fe551ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.751 INFO:teuthology.orchestra.run.smithi082.stdout:5 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 10 librados: watch_flush enter 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 10 librados: watch_flush exit 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 10 client.4149.objecter successfully canceled tick 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 10 monclient: shutdown 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 >> v1:172.21.15.79:6789/0 conn(0x7fe55c145a10 legacy=0x7fe55c0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 shutdown_connections 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 >> 172.21.15.82:0/932536967 conn(0x7fe55c066b00 msgr2=0x7fe55c066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 shutdown_connections 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 1 -- 172.21.15.82:0/932536967 wait complete. 2024-03-29T10:47:50.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.769+0000 7fe5632a8640 1 librados: shutdown 2024-03-29T10:47:50.761 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 00c2423f-5d63-4648-a44b-2fadd7698e00 6 2024-03-29T10:47:50.897 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:50.915+0000 7feb80432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:47:50.900 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.919+0000 7f2f11a81640 10 monclient: get_monmap_and_config 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.919+0000 7f2f11a81640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.919+0000 7f2f11a81640 10 monclient: monmap: 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:50.919966+0000 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:50.919966+0000 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:50.901 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.919+0000 7f2f11a81640 1 Processor -- start 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.919+0000 7f2f11a81640 1 -- start start 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.919+0000 7f2f11a81640 10 monclient: init 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient(hunting): picked mon.noname-b con 0x7f2f0c09ac40 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient(hunting): picked mon.noname-c con 0x7f2f0c09c870 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0c0b4360 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c09dec0 con 0x7f2f0c0b4360 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c09da30 con 0x7f2f0c09c870 2024-03-29T10:47:50.911 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c0b39c0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f11a81640 10 monclient(hunting): authenticate will time out at 833.091707s 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0affd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f2f0c09c870 0x7f2f0c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:38722/0 (socket says 172.21.15.82:38722) 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0affd640 1 -- 172.21.15.82:0/4074517616 learned_addr learned my addr 172.21.15.82:0/4074517616 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930043260 0 0) 0x7f2f0c09da30 con 0x7f2f0c09c870 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef4003520 con 0x7f2f0c09c870 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630176548 0 0) 0x7f2f04001110 con 0x7f2f0c0b4360 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.912 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c09da30 con 0x7f2f0c0b4360 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739186885 0 0) 0x7f2f0c0b39c0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.920+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f04001110 con 0x7f2f0c09ac40 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452683963 0 0) 0x7f2ef4003520 con 0x7f2f0c09c870 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 >> v1:172.21.15.79:6790/0 conn(0x7f2f0c09ac40 legacy=0x7f2f0c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 >> v1:172.21.15.79:6789/0 conn(0x7f2f0c0b4360 legacy=0x7f2f0c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0b5160 con 0x7f2f0c09c870 2024-03-29T10:47:50.913 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2f00001690 con 0x7f2f0c09c870 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: dump: 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.914 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f000017f0 con 0x7f2f0c09c870 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 1 -- 172.21.15.82:0/4074517616 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2f00001c40 con 0x7f2f0c09c870 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 4 set_mon_vals no callback set 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f0a7fc640 10 monclient: dump: 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.915 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 10 monclient: shutdown 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 1 -- 172.21.15.82:0/4074517616 >> v1:172.21.15.82:6789/0 conn(0x7f2f0c09c870 legacy=0x7f2f0c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 1 -- 172.21.15.82:0/4074517616 shutdown_connections 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 1 -- 172.21.15.82:0/4074517616 >> 172.21.15.82:0/4074517616 conn(0x7f2f0c066b00 msgr2=0x7f2f0c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 1 -- 172.21.15.82:0/4074517616 shutdown_connections 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.921+0000 7f2f11a81640 1 -- 172.21.15.82:0/4074517616 wait complete. 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.922+0000 7f2f11a81640 10 monclient: build_initial_monmap 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.922+0000 7f2f11a81640 10 monclient: monmap: 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:50.916 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.922+0000 7f2f11a81640 1 librados: starting msgr at 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.922+0000 7f2f11a81640 1 librados: starting objecter 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.922+0000 7f2f11a81640 1 Processor -- start 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 1 -- start start 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 1 librados: setting wanted keys 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 1 librados: calling monclient init 2024-03-29T10:47:50.917 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient: init 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient(hunting): picked mon.noname-a con 0x7f2f0c09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient(hunting): picked mon.noname-b con 0x7f2f0c0b1110 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient(hunting): picked mon.noname-c con 0x7f2f0c1d7940 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c0b15a0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c1d6d30 con 0x7f2f0c0b1110 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0c13ffa0 con 0x7f2f0c1d7940 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f11a81640 10 monclient(hunting): authenticate will time out at 833.094907s 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f0affd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f2f0c0b1110 0x7f2f0c1d7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:38732/0 (socket says 172.21.15.82:38732) 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2f0affd640 1 -- 172.21.15.82:0/3731857177 learned_addr learned my addr 172.21.15.82:0/3731857177 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 10 client.?.objecter ms_handle_connect 0x7f2f0c0b1110 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.918 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 10 client.?.objecter ms_handle_connect 0x7f2f0c09ac40 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 10 client.?.objecter ms_handle_connect 0x7f2f0c1d7940 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.923+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777780986 0 0) 0x7f2f0c1d6d30 con 0x7f2f0c0b1110 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8004eb0 con 0x7f2f0c0b1110 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039293273 0 0) 0x7f2f04001110 con 0x7f2f0c09ac40 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.919 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c1d6d30 con 0x7f2f0c09ac40 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624549754 0 0) 0x7f2f0c13ffa0 con 0x7f2f0c1d7940 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f04001110 con 0x7f2f0c1d7940 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275537535 0 0) 0x7f2f0c1d6d30 con 0x7f2f0c09ac40 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f0c13ffa0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2728878859 0 0) 0x7f2ef8004eb0 con 0x7f2f0c0b1110 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f0c1d6d30 con 0x7f2f0c0b1110 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2efc001a50 con 0x7f2f0c09ac40 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): dump: 2024-03-29T10:47:50.920 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2f00001b00 con 0x7f2f0c0b1110 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): dump: 2024-03-29T10:47:50.921 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813715914 0 0) 0x7f2f04001110 con 0x7f2f0c1d7940 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ef8004eb0 con 0x7f2f0c1d7940 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2f04001960 con 0x7f2f0c1d7940 2024-03-29T10:47:50.922 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient(hunting): dump: 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1344479795 0 0) 0x7f2f0c13ffa0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 >> v1:172.21.15.79:6790/0 conn(0x7f2f0c1d7940 legacy=0x7f2f0c1e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 >> v1:172.21.15.82:6789/0 conn(0x7f2f0c0b1110 legacy=0x7f2f0c1d7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.923 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 monclient: found mon.a 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0ad870 con 0x7f2f0c09ac40 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 5 monclient: authenticate success, global_id 4155 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 10 monclient: _renew_subs 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c1e2f70 con 0x7f2f0c09ac40 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 10 monclient: _renew_subs 2024-03-29T10:47:50.924 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c1e32e0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f11a81640 1 librados: init done 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2efc001bb0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2efc003b30 con 0x7f2f0c09ac40 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2f09ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 monclient: dump: 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:50.925 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2efc004020 con 0x7f2f0c09ac40 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2627756591 0 0) 0x7f2efc0048c0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 10 client.4155.objecter ms_dispatch 0x7f2f0c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.924+0000 7f2ef3fff640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.925+0000 7f2f11a81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.925+0000 7f2f11a81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.925+0000 7f2f11a81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.925+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec005080 con 0x7f2f0c09ac40 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.927+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2efc004bb0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.926 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.927+0000 7f2ef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:50.927 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.927+0000 7f2ef3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:50.970 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:50.988+0000 7f793a01e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:47:50.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.998+0000 7f2f11a81640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "00c2423f-5d63-4648-a44b-2fadd7698e00", "id": 6}] 2024-03-29T10:47:50.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.999+0000 7f2f11a81640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "00c2423f-5d63-4648-a44b-2fadd7698e00", "id": 6}] 2024-03-29T10:47:50.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.999+0000 7f2f11a81640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:50.981 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:50.999+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd new", "uuid": "00c2423f-5d63-4648-a44b-2fadd7698e00", "id": 6} v 0) v1 -- 0x7f2eec0058b0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.000+0000 7f2ef3fff640 1 -- 172.21.15.82:0/3731857177 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00c2423f-5d63-4648-a44b-2fadd7698e00", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2756829723 0 1113811849) 0x7f2efc0042d0 con 0x7f2f0c09ac40 2024-03-29T10:47:50.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.000+0000 7f2ef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "00c2423f-5d63-4648-a44b-2fadd7698e00", "id": 6}] 2024-03-29T10:47:50.982 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.000+0000 7f2ef3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:50.983 INFO:teuthology.orchestra.run.smithi082.stdout:6 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 10 librados: watch_flush enter 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 10 librados: watch_flush exit 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 10 client.4155.objecter successfully canceled tick 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 10 monclient: shutdown 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:50.984 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 >> v1:172.21.15.79:6789/0 conn(0x7f2f0c09ac40 legacy=0x7f2f0c0b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:50.985 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 shutdown_connections 2024-03-29T10:47:50.985 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 >> 172.21.15.82:0/3731857177 conn(0x7f2f0c066b00 msgr2=0x7f2f0c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:50.985 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 shutdown_connections 2024-03-29T10:47:50.985 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 1 -- 172.21.15.82:0/3731857177 wait complete. 2024-03-29T10:47:50.985 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.001+0000 7f2f11a81640 1 librados: shutdown 2024-03-29T10:47:50.994 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new ae0947ca-ffcb-4179-8a37-b246ab449ff4 7 2024-03-29T10:47:51.034 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.052+0000 7feb80432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.109+0000 7f9763fd6640 10 monclient: get_monmap_and_config 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.109+0000 7f9763fd6640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.109+0000 7f9763fd6640 10 monclient: monmap: 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:51.110797+0000 2024-03-29T10:47:51.092 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:51.110797+0000 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.110+0000 7f9763fd6640 1 Processor -- start 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.110+0000 7f9763fd6640 1 -- start start 2024-03-29T10:47:51.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.110+0000 7f9763fd6640 10 monclient: init 2024-03-29T10:47:51.101 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f975c09ac40 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f975c09c870 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f975c0b4360 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c09dec0 con 0x7f975c0b4360 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c09da30 con 0x7f975c09ac40 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0b39c0 con 0x7f975c09c870 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 10 monclient(hunting): authenticate will time out at 833.282639s 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9761d72640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f975c09ac40 0x7f975c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:38748/0 (socket says 172.21.15.82:38748) 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9761d72640 1 -- 172.21.15.82:0/4087003202 learned_addr learned my addr 172.21.15.82:0/4087003202 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:51.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904903311 0 0) 0x7f975c09da30 con 0x7f975c09ac40 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9748003520 con 0x7f975c09ac40 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127230773 0 0) 0x7f9754001490 con 0x7f975c0b4360 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c09da30 con 0x7f975c0b4360 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980215559 0 0) 0x7f975c0b39c0 con 0x7f975c09c870 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9754001490 con 0x7f975c09c870 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3318670864 0 0) 0x7f9748003520 con 0x7f975c09ac40 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 >> v1:172.21.15.79:6790/0 conn(0x7f975c09c870 legacy=0x7f975c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 >> v1:172.21.15.79:6789/0 conn(0x7f975c0b4360 legacy=0x7f975c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 monclient: found mon.noname-c 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c0b5160 con 0x7f975c09ac40 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.104 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9763fd6640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.111+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f974c001690 con 0x7f975c09ac40 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: dump: 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.105 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f974c001870 con 0x7f975c09ac40 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 1 -- 172.21.15.82:0/4087003202 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f974c001c80 con 0x7f975c09ac40 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 4 set_mon_vals no callback set 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9760d70640 10 monclient: dump: 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.106 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 10 monclient: shutdown 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 1 -- 172.21.15.82:0/4087003202 >> v1:172.21.15.82:6789/0 conn(0x7f975c09ac40 legacy=0x7f975c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 1 -- 172.21.15.82:0/4087003202 shutdown_connections 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 1 -- 172.21.15.82:0/4087003202 >> 172.21.15.82:0/4087003202 conn(0x7f975c066b00 msgr2=0x7f975c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 1 -- 172.21.15.82:0/4087003202 shutdown_connections 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.112+0000 7f9763fd6640 1 -- 172.21.15.82:0/4087003202 wait complete. 2024-03-29T10:47:51.107 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 10 monclient: monmap: 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 0 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 0.000000 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:created 0.000000 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 1 librados: starting msgr at 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 1 librados: starting objecter 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 1 Processor -- start 2024-03-29T10:47:51.108 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 1 -- start start 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 1 librados: setting wanted keys 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 1 librados: calling monclient init 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.113+0000 7f9763fd6640 10 monclient: init 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f975c09ac40 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f975c0b5e30 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f975c141ec0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0acfc0 con 0x7f975c09ac40 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c1d72c0 con 0x7f975c0b5e30 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c140cd0 con 0x7f975c141ec0 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9763fd6640 10 monclient(hunting): authenticate will time out at 833.285499s 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9761571640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f975c0b5e30 0x7f975c1417b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:38760/0 (socket says 172.21.15.82:38760) 2024-03-29T10:47:51.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f9761571640 1 -- 172.21.15.82:0/412485669 learned_addr learned my addr 172.21.15.82:0/412485669 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 client.?.objecter ms_handle_connect 0x7f975c0b5e30 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 client.?.objecter ms_handle_connect 0x7f975c09ac40 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 client.?.objecter ms_handle_connect 0x7f975c141ec0 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812665232 0 0) 0x7f975c1d72c0 con 0x7f975c0b5e30 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9740004eb0 con 0x7f975c0b5e30 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184047977 0 0) 0x7f975c0acfc0 con 0x7f975c09ac40 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.110 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c1d72c0 con 0x7f975c09ac40 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341059336 0 0) 0x7f975c140cd0 con 0x7f975c141ec0 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c0acfc0 con 0x7f975c141ec0 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580781833 0 0) 0x7f9740004eb0 con 0x7f975c0b5e30 2024-03-29T10:47:51.111 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f975c140cd0 con 0x7f975c0b5e30 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9758001980 con 0x7f975c0b5e30 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): dump: 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.112 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158184198 0 0) 0x7f975c1d72c0 con 0x7f975c09ac40 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9740004eb0 con 0x7f975c09ac40 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f974c001b80 con 0x7f975c09ac40 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): dump: 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.113 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 984203551 0 0) 0x7f975c0acfc0 con 0x7f975c141ec0 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f975c1d72c0 con 0x7f975c141ec0 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9754001bc0 con 0x7f975c141ec0 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 10 monclient(hunting): dump: 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.114 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3309485983 0 0) 0x7f975c140cd0 con 0x7f975c0b5e30 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.114+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 >> v1:172.21.15.79:6790/0 conn(0x7f975c141ec0 legacy=0x7f975c1e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 >> v1:172.21.15.79:6789/0 conn(0x7f975c09ac40 legacy=0x7f975c1d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 monclient: found mon.b 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c1d7ef0 con 0x7f975c0b5e30 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:47:51.115 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 monclient: _renew_subs 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975c1e3280 con 0x7f975c0b5e30 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 monclient: _renew_subs 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f975c1e35f0 con 0x7f975c0b5e30 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 1 librados: init done 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9758001c40 con 0x7f975c0b5e30 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9758003a60 con 0x7f975c0b5e30 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 monclient: dump: 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:epoch 1 2024-03-29T10:47:51.116 INFO:teuthology.orchestra.run.smithi082.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:election_strategy: 1 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9753fff640 4 set_mon_vals no callback set 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9758003f50 con 0x7f975c0b5e30 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2966374911 0 0) 0x7f9758005900 con 0x7f975c0b5e30 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 10 client.4144.objecter ms_dispatch 0x7f975c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 3 client.4144.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 3 client.4144.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:47:51.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f97527fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.115+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c09b020 con 0x7f975c0b5e30 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.117+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f97580043e0 con 0x7f975c0b5e30 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.117+0000 7f97527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.118 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.117+0000 7f97527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:51.118 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.129+0000 7f793a01e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:47:51.150 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.168+0000 7feb80432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:47:51.171 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.189+0000 7f9763fd6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae0947ca-ffcb-4179-8a37-b246ab449ff4", "id": 7}] 2024-03-29T10:47:51.171 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.189+0000 7f9763fd6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae0947ca-ffcb-4179-8a37-b246ab449ff4", "id": 7}] 2024-03-29T10:47:51.171 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.189+0000 7f9763fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:51.171 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.189+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ae0947ca-ffcb-4179-8a37-b246ab449ff4", "id": 7} v 0) v1 -- 0x7f975c067080 con 0x7f975c0b5e30 2024-03-29T10:47:51.175 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.193+0000 7f97527fc640 1 -- 172.21.15.82:0/412485669 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae0947ca-ffcb-4179-8a37-b246ab449ff4", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2737837728 0 1371665406) 0x7f9758005ac0 con 0x7f975c0b5e30 2024-03-29T10:47:51.175 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.193+0000 7f97527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae0947ca-ffcb-4179-8a37-b246ab449ff4", "id": 7}] 2024-03-29T10:47:51.175 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.193+0000 7f97527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:51.175 INFO:teuthology.orchestra.run.smithi082.stdout:7 2024-03-29T10:47:51.176 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 10 librados: watch_flush enter 2024-03-29T10:47:51.176 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 10 librados: watch_flush exit 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 10 client.4144.objecter successfully canceled tick 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 10 monclient: shutdown 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 >> v1:172.21.15.82:6789/0 conn(0x7f975c0b5e30 legacy=0x7f975c1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 shutdown_connections 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 >> 172.21.15.82:0/412485669 conn(0x7f975c066b00 msgr2=0x7f975c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 shutdown_connections 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 1 -- 172.21.15.82:0/412485669 wait complete. 2024-03-29T10:47:51.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-29T10:47:51.194+0000 7f9763fd6640 1 librados: shutdown 2024-03-29T10:47:51.186 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:47:51.186 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:47:51.188 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:47:51.188 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:47:51.188 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:47:51.190 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:47:51.190 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:47:51.191 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:47:51.192 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:47:51.193 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:47:51.193 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:47:51.195 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:47:51.195 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:47:51.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:47:51.197 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:47:51.197 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:47:51.197 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:47:51.198 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.216+0000 7feb80432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:47:51.199 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:47:51.199 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:47:51.200 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:47:51.201 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:47:51.202 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:47:51.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:47:51.204 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:47:51.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:51.237 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.255+0000 7f793a01e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:47:51.302 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.320+0000 7f793a01e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:47:51.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.347+0000 7fb0ae390640 10 monclient: get_monmap_and_config 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.347+0000 7fb0ae390640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.347+0000 7fb0ae390640 10 monclient: monmap: 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:51.348567+0000 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:51.348567+0000 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:51.330 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 1 Processor -- start 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 1 -- start start 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 10 monclient: init 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 10 monclient(hunting): picked mon.noname-b con 0x7fb0a80b8240 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.348+0000 7fb0ae390640 10 monclient(hunting): picked mon.noname-c con 0x7fb0a80b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 10 monclient(hunting): picked mon.noname-a con 0x7fb0a809c8c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:51.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a809bd80 con 0x7fb0a809c8c0 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ac92d640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fb0a809c8c0 0x7fb0a814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:43586/0 (socket says 172.21.15.79:43586) 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ac92d640 1 -- 172.21.15.79:0/1644224117 learned_addr learned my addr 172.21.15.79:0/1644224117 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a809b960 con 0x7fb0a80b1f60 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a80b2ec0 con 0x7fb0a80b8240 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 10 monclient(hunting): authenticate will time out at 990.641338s 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946854248 0 0) 0x7fb0a809bd80 con 0x7fb0a809c8c0 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb090003520 con 0x7fb0a809c8c0 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351529090 0 0) 0x7fb0a80b2ec0 con 0x7fb0a80b8240 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): my global_id is 4175 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a809bd80 con 0x7fb0a80b8240 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240738333 0 0) 0x7fb0a809b960 con 0x7fb0a80b1f60 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient(hunting): my global_id is 4168 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a80b2ec0 con 0x7fb0a80b1f60 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3056193203 0 0) 0x7fb090003520 con 0x7fb0a809c8c0 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 >> v1:172.21.15.79:6790/0 conn(0x7fb0a80b8240 legacy=0x7fb0a813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 >> v1:172.21.15.82:6789/0 conn(0x7fb0a80b1f60 legacy=0x7fb0a809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 monclient: found mon.noname-a 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a813fb60 con 0x7fb0a809c8c0 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb0a0001a10 con 0x7fb0a809c8c0 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: dump: 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.342 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0a0001b70 con 0x7fb0a809c8c0 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0ae390640 4 set_mon_vals no callback set 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 1 -- 172.21.15.79:0/1644224117 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb0a0001fc0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.349+0000 7fb0a6ffd640 10 monclient: dump: 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 10 monclient: shutdown 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 >> v1:172.21.15.79:6789/0 conn(0x7fb0a809c8c0 legacy=0x7fb0a814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 shutdown_connections 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 >> 172.21.15.79:0/1644224117 conn(0x7fb0a8066b00 msgr2=0x7fb0a8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 shutdown_connections 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.350+0000 7fb0ae390640 1 -- 172.21.15.79:0/1644224117 wait complete. 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 10 monclient: monmap: 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 0 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 0.000000 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:created 0.000000 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:51.345 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 1 librados: starting msgr at 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 1 librados: starting objecter 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 1 Processor -- start 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 1 -- start start 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 1 librados: setting wanted keys 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 1 librados: calling monclient init 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.351+0000 7fb0ae390640 10 monclient: init 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient(hunting): picked mon.noname-b con 0x7fb0a809c8c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient(hunting): picked mon.noname-a con 0x7fb0a80b8240 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:51.346 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a77fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fb0a80b8240 0x7fb0a8143ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:43602/0 (socket says 172.21.15.79:43602) 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a77fe640 1 -- 172.21.15.79:0/2524191733 learned_addr learned my addr 172.21.15.79:0/2524191733 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient(hunting): picked mon.noname-c con 0x7fb0a81335f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a81435b0 con 0x7fb0a80b8240 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a81445f0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb0a80b8240 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb0a81335f0 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a81437e0 con 0x7fb0a81335f0 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0ae390640 10 monclient(hunting): authenticate will time out at 990.644723s 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278773787 0 0) 0x7fb0a81435b0 con 0x7fb0a80b8240 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.347 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb094004eb0 con 0x7fb0a80b8240 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb0a809c8c0 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022544233 0 0) 0x7fb0a81437e0 con 0x7fb0a81335f0 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): my global_id is 4178 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a81435b0 con 0x7fb0a81335f0 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960151422 0 0) 0x7fb0a81445f0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.348 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 10 monclient(hunting): my global_id is 4171 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a81437e0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159234098 0 0) 0x7fb094004eb0 con 0x7fb0a80b8240 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.352+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb094005670 con 0x7fb0a80b8240 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3663778854 0 0) 0x7fb0a81435b0 con 0x7fb0a81335f0 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb094004eb0 con 0x7fb0a81335f0 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb098001950 con 0x7fb0a80b8240 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.349 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb0a0001e80 con 0x7fb0a81335f0 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.350 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3758216556 0 0) 0x7fb0a81437e0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0a81435b0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb09c0015e0 con 0x7fb0a809c8c0 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.351 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3289801675 0 0) 0x7fb094005670 con 0x7fb0a80b8240 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 >> v1:172.21.15.79:6790/0 conn(0x7fb0a81335f0 legacy=0x7fb0a81e1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 >> v1:172.21.15.82:6789/0 conn(0x7fb0a809c8c0 legacy=0x7fb0a81419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 monclient: found mon.a 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a8143ac0 con 0x7fb0a80b8240 2024-03-29T10:47:51.352 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 10 monclient: _renew_subs 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a813c840 con 0x7fb0a80b8240 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 10 monclient: _renew_subs 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a8132c10 con 0x7fb0a80b8240 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0ae390640 1 librados: init done 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb098001ab0 con 0x7fb0a80b8240 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb0980038a0 con 0x7fb0a80b8240 2024-03-29T10:47:51.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 monclient: dump: 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:epoch 1 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb098003e50 con 0x7fb0a80b8240 2024-03-29T10:47:51.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fb0980068e0 con 0x7fb0a80b8240 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 10 client.4191.objecter ms_dispatch 0x7fb0a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a67fc640 4 set_mon_vals no callback set 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.353+0000 7fb0a4ff9640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.354+0000 7fb0ae390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.354+0000 7fb0ae390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.354+0000 7fb0ae390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.354+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb088005080 con 0x7fb0a80b8240 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.356+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb098004330 con 0x7fb0a80b8240 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.356+0000 7fb0a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.356+0000 7fb0a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:51.356 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.358+0000 7feb80432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:51.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0ae390640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0ae390640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0ae390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:47:51.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb088005330 con 0x7fb0a80b8240 2024-03-29T10:47:51.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0a4ff9640 1 -- 172.21.15.79:0/2524191733 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fb098006af0 con 0x7fb0a80b8240 2024-03-29T10:47:51.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.417+0000 7fb0a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:51.400 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-03-29T10:47:51.400 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:51.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 10 librados: watch_flush enter 2024-03-29T10:47:51.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 10 librados: watch_flush exit 2024-03-29T10:47:51.401 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 10 monclient: shutdown 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 >> v1:172.21.15.79:6789/0 conn(0x7fb0a80b8240 legacy=0x7fb0a8143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 shutdown_connections 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 >> 172.21.15.79:0/2524191733 conn(0x7fb0a8066b00 msgr2=0x7fb0a813dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 shutdown_connections 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 1 -- 172.21.15.79:0/2524191733 wait complete. 2024-03-29T10:47:51.402 INFO:teuthology.orchestra.run.smithi079.stderr:2024-03-29T10:47:51.418+0000 7fb0ae390640 1 librados: shutdown 2024-03-29T10:47:51.410 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:47:51.411 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:47:51.413 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.431+0000 7f793a01e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:51.440 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.458+0000 7feb80432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:47:51.506 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.524+0000 7f793a01e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:47:51.597 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.615+0000 7feb80432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:51.640 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.658+0000 7feb80432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:51.660 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.678+0000 7f793a01e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:47:51.701 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.719+0000 7f793a01e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:47:51.794 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-29T10:47:51.812+0000 7fba7fc16640 -1 Falling back to public interface 2024-03-29T10:47:51.795 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:51.813+0000 7feb80432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:47:51.801 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-29T10:47:51.819+0000 7ff3116d9640 -1 Falling back to public interface 2024-03-29T10:47:51.802 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-29T10:47:51.819+0000 7ffa57ed9640 -1 Falling back to public interface 2024-03-29T10:47:51.803 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-29T10:47:51.819+0000 7ffa56e88640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T10:47:51.803 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-29T10:47:51.820+0000 7ffa56e88640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6807/0: (98) Address already in use 2024-03-29T10:47:51.808 INFO:tasks.ceph.osd.1.smithi079.stderr:2024-03-29T10:47:51.826+0000 7f76a7b19640 -1 Falling back to public interface 2024-03-29T10:47:51.813 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-29T10:47:51.831+0000 7fc158c10640 -1 Falling back to public interface 2024-03-29T10:47:51.817 INFO:tasks.ceph.osd.0.smithi079.stderr:2024-03-29T10:47:51.835+0000 7fb00901b640 -1 Falling back to public interface 2024-03-29T10:47:51.820 INFO:tasks.ceph.osd.2.smithi079.stderr:2024-03-29T10:47:51.838+0000 7f1436c13640 -1 Falling back to public interface 2024-03-29T10:47:51.825 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-03-29T10:47:51.842+0000 7fc36fab5640 -1 Falling back to public interface 2024-03-29T10:47:51.856 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:51.874+0000 7f793a01e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:47:52.216 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:52.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.322+0000 7f19c6eb3640 10 monclient: get_monmap_and_config 2024-03-29T10:47:52.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.322+0000 7f19c6eb3640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.322+0000 7f19c6eb3640 10 monclient: monmap: 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:52.323561+0000 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:52.323561+0000 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:52.305 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.323+0000 7f19c6eb3640 1 Processor -- start 2024-03-29T10:47:52.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.323+0000 7f19c6eb3640 1 -- start start 2024-03-29T10:47:52.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.323+0000 7f19c6eb3640 10 monclient: init 2024-03-29T10:47:52.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.323+0000 7f19c6eb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.323+0000 7f19c6eb3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.323+0000 7f19c6eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f19c01457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f19c013f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f19c00a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c00aaef0 con 0x7f19c013f650 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c0145bd0 con 0x7f19c01457f0 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c0145eb0 con 0x7f19c00a9d70 2024-03-29T10:47:52.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 10 monclient(hunting): authenticate will time out at 991.616173s 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b7fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f19c013f650 0x7f19c00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:43604/0 (socket says 172.21.15.79:43604) 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c5450640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f19c00a9d70 0x7f19c014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:36504/0 (socket says 172.21.15.79:36504) 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b7fff640 1 -- 172.21.15.79:0/2948376418 learned_addr learned my addr 172.21.15.79:0/2948376418 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223777023 0 0) 0x7f19c00aaef0 con 0x7f19c013f650 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a8003520 con 0x7f19c013f650 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223263087 0 0) 0x7f19c0145eb0 con 0x7f19c00a9d70 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c00aaef0 con 0x7f19c00a9d70 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144681548 0 0) 0x7f19c00aaef0 con 0x7f19c00a9d70 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 >> v1:172.21.15.82:6789/0 conn(0x7f19c01457f0 legacy=0x7f19c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 >> v1:172.21.15.79:6789/0 conn(0x7f19c013f650 legacy=0x7f19c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 monclient: found mon.noname-b 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0146340 con 0x7f19c00a9d70 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f19b80019f0 con 0x7f19c00a9d70 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 5 monclient: authenticate success, global_id 4181 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:52.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: dump: 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b8001b50 con 0x7f19c00a9d70 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 1 -- 172.21.15.79:0/2948376418 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f19b8001fa0 con 0x7f19c00a9d70 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:52.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19c6eb3640 4 set_mon_vals no callback set 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.324+0000 7f19b77fe640 10 monclient: dump: 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 10 monclient: shutdown 2024-03-29T10:47:52.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 1 -- 172.21.15.79:0/2948376418 >> v1:172.21.15.79:6790/0 conn(0x7f19c00a9d70 legacy=0x7f19c014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 1 -- 172.21.15.79:0/2948376418 shutdown_connections 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 1 -- 172.21.15.79:0/2948376418 >> 172.21.15.79:0/2948376418 conn(0x7f19c0066a00 msgr2=0x7f19c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 1 -- 172.21.15.79:0/2948376418 shutdown_connections 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 1 -- 172.21.15.79:0/2948376418 wait complete. 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 10 monclient: build_initial_monmap 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.325+0000 7f19c6eb3640 10 monclient: monmap: 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:52.325 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 librados: starting msgr at 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 librados: starting objecter 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 Processor -- start 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 -- start start 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 librados: setting wanted keys 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 librados: calling monclient init 2024-03-29T10:47:52.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient: init 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f19c00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f19c013f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f19c01457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c0141b10 con 0x7f19c01457f0 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c013fda0 con 0x7f19c00a9d70 2024-03-29T10:47:52.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c0140080 con 0x7f19c013f650 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.326+0000 7f19c6eb3640 10 monclient(hunting): authenticate will time out at 991.619101s 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c5450640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f19c01457f0 0x7f19c0142740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:43616/0 (socket says 172.21.15.79:43616) 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c5450640 1 -- 172.21.15.79:0/1931699079 learned_addr learned my addr 172.21.15.79:0/1931699079 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.?.objecter ms_handle_connect 0x7f19c01457f0 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.?.objecter ms_handle_connect 0x7f19c013f650 2024-03-29T10:47:52.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.?.objecter ms_handle_connect 0x7f19c00a9d70 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133144898 0 0) 0x7f19c0140080 con 0x7f19c013f650 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac004eb0 con 0x7f19c013f650 2024-03-29T10:47:52.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397609278 0 0) 0x7f19c0141b10 con 0x7f19c01457f0 2024-03-29T10:47:52.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:47:52.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c0140080 con 0x7f19c01457f0 2024-03-29T10:47:52.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599935663 0 0) 0x7f19c013fda0 con 0x7f19c00a9d70 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): my global_id is 4177 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c0141b10 con 0x7f19c00a9d70 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2151700411 0 0) 0x7f19ac004eb0 con 0x7f19c013f650 2024-03-29T10:47:52.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c013fda0 con 0x7f19c013f650 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f19b00012d0 con 0x7f19c013f650 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.332 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 435592666 0 0) 0x7f19c0140080 con 0x7f19c01457f0 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19ac004eb0 con 0x7f19c01457f0 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f19b80019b0 con 0x7f19c01457f0 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:52.334 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:52.335 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2427339521 0 0) 0x7f19c0141b10 con 0x7f19c00a9d70 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c0140080 con 0x7f19c00a9d70 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f19bc001750 con 0x7f19c00a9d70 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:52.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient(hunting): dump: 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.337 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232147450 0 0) 0x7f19c013fda0 con 0x7f19c013f650 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 >> v1:172.21.15.82:6789/0 conn(0x7f19c00a9d70 legacy=0x7f19c01d71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 >> v1:172.21.15.79:6789/0 conn(0x7f19c01457f0 legacy=0x7f19c0142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 monclient: found mon.c 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:47:52.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c013b560 con 0x7f19c013f650 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 5 monclient: authenticate success, global_id 4184 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 10 monclient: _renew_subs 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c013b810 con 0x7f19c013f650 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 10 monclient: _renew_subs 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c01d45b0 con 0x7f19c013f650 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19c6eb3640 1 librados: init done 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b00047e0 con 0x7f19c013f650 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:52.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f19b0004af0 con 0x7f19c013f650 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b6ffd640 4 set_mon_vals no callback set 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 monclient: dump: 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f19b0004fe0 con 0x7f19c013f650 2024-03-29T10:47:52.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f19b0006850 con 0x7f19c013f650 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 10 client.4184.objecter ms_dispatch 0x7f19c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.327+0000 7f19b57fa640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.328+0000 7f19b57fa640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.328+0000 7f19c6eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.328+0000 7f19c6eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.328+0000 7f19c6eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.328+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a0005080 con 0x7f19c013f650 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.330+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f19b0006a90 con 0x7f19c013f650 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.330+0000 7f19b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:52.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.330+0000 7f19b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:52.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.406+0000 7f19c6eb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:52.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.406+0000 7f19c6eb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:52.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.406+0000 7f19c6eb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:47:52.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.406+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19a0005330 con 0x7f19c013f650 2024-03-29T10:47:52.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19b57fa640 1 -- 172.21.15.79:0/1931699079 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f19b0006340 con 0x7f19c013f650 2024-03-29T10:47:52.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:52.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:52.389 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:47:52.390 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:52.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 10 librados: watch_flush enter 2024-03-29T10:47:52.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 10 librados: watch_flush exit 2024-03-29T10:47:52.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T10:47:52.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 10 client.4184.objecter successfully canceled tick 2024-03-29T10:47:52.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 10 monclient: shutdown 2024-03-29T10:47:52.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:52.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.407+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 >> v1:172.21.15.79:6790/0 conn(0x7f19c013f650 legacy=0x7f19c01405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:52.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.408+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 shutdown_connections 2024-03-29T10:47:52.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.408+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 >> 172.21.15.79:0/1931699079 conn(0x7f19c0066a00 msgr2=0x7f19c00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:52.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.408+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 shutdown_connections 2024-03-29T10:47:52.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.408+0000 7f19c6eb3640 1 -- 172.21.15.79:0/1931699079 wait complete. 2024-03-29T10:47:52.393 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:52.408+0000 7f19c6eb3640 1 librados: shutdown 2024-03-29T10:47:52.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:52.620 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:52.638+0000 7feb80432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:47:52.640 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:52.658+0000 7f793a01e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:47:52.892 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:52.910+0000 7feb80432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:47:52.910 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:52.928+0000 7f793a01e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:47:52.946 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:52.964+0000 7feb80432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:47:52.962 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:52.980+0000 7f793a01e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:47:53.061 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:53.079+0000 7feb80432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:47:53.076 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-29T10:47:53.094+0000 7f793a01e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:47:54.077 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:54.095+0000 7feb209ae640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:47:54.078 INFO:tasks.ceph.mgr.y.smithi079.stderr:2024-03-29T10:47:54.096+0000 7feb209ae640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:47:55.280 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-29T10:47:55.298+0000 7ffa57ed9640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:47:55.304 INFO:tasks.ceph.osd.1.smithi079.stderr:2024-03-29T10:47:55.321+0000 7f76a7b19640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:47:55.313 INFO:tasks.ceph.osd.0.smithi079.stderr:2024-03-29T10:47:55.331+0000 7fb00901b640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:47:55.332 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-29T10:47:55.350+0000 7fba7fc16640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:47:55.333 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-29T10:47:55.350+0000 7ff3116d9640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:47:55.335 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-29T10:47:55.353+0000 7fc158c10640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:47:55.361 INFO:tasks.ceph.osd.2.smithi079.stderr:2024-03-29T10:47:55.379+0000 7f1436c13640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:47:55.372 INFO:tasks.ceph.osd.3.smithi079.stderr:2024-03-29T10:47:55.390+0000 7fc36fab5640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:47:59.206 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.386+0000 7fbd1e006640 10 monclient: get_monmap_and_config 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.386+0000 7fbd1e006640 10 monclient: build_initial_monmap 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.386+0000 7fbd1e006640 10 monclient: monmap: 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:59.387826+0000 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:59.387826+0000 2024-03-29T10:47:59.369 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:47:59.370 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.387+0000 7fbd1e006640 1 Processor -- start 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.388+0000 7fbd1e006640 1 -- start start 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.388+0000 7fbd1e006640 10 monclient: init 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient(hunting): picked mon.noname-c con 0x7fbd181422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:59.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient(hunting): picked mon.noname-a con 0x7fbd18142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient(hunting): picked mon.noname-b con 0x7fbd180b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18142c80 con 0x7fbd18142690 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180aae40 con 0x7fbd181422b0 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1813fbd0 con 0x7fbd180b17d0 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:59.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd1e006640 10 monclient(hunting): authenticate will time out at 998.681341s 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd177fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fbd181422b0 0x7fbd1813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:58524/0 (socket says 172.21.15.79:58524) 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd177fe640 1 -- 172.21.15.79:0/292982959 learned_addr learned my addr 172.21.15.79:0/292982959 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.389+0000 7fbd17fff640 1 --1- 172.21.15.79:0/292982959 >> v1:172.21.15.79:6790/0 conn(0x7fbd180b17d0 0x7fbd180b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:53632/0 (socket says 172.21.15.79:53632) 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584212996 0 0) 0x7fbd18142c80 con 0x7fbd18142690 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd00003520 con 0x7fbd18142690 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958435368 0 0) 0x7fbd180aae40 con 0x7fbd181422b0 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient(hunting): my global_id is 4219 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd18142c80 con 0x7fbd181422b0 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3151430913 0 0) 0x7fbd00003520 con 0x7fbd18142690 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 >> v1:172.21.15.79:6790/0 conn(0x7fbd180b17d0 legacy=0x7fbd180b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 >> v1:172.21.15.82:6789/0 conn(0x7fbd181422b0 legacy=0x7fbd1813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 monclient: found mon.noname-a 2024-03-29T10:47:59.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18151940 con 0x7fbd18142690 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbd0c001690 con 0x7fbd18142690 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd167fc640 10 monclient: dump: 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.375 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd1e006640 5 monclient: authenticate success, global_id 4242 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.390+0000 7fbd1e006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd0c0017f0 con 0x7fbd18142690 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd1e006640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd1e006640 4 set_mon_vals no callback set 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd167fc640 1 -- 172.21.15.79:0/292982959 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbd0c001bc0 con 0x7fbd18142690 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.391+0000 7fbd167fc640 10 monclient: dump: 2024-03-29T10:47:59.376 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:59.377 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:59.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.392+0000 7fbd1e006640 10 monclient: shutdown 2024-03-29T10:47:59.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.392+0000 7fbd1e006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.392+0000 7fbd1e006640 1 -- 172.21.15.79:0/292982959 >> v1:172.21.15.79:6789/0 conn(0x7fbd18142690 legacy=0x7fbd180b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.393+0000 7fbd1e006640 1 -- 172.21.15.79:0/292982959 shutdown_connections 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.393+0000 7fbd1e006640 1 -- 172.21.15.79:0/292982959 >> 172.21.15.79:0/292982959 conn(0x7fbd18066a00 msgr2=0x7fbd18066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.393+0000 7fbd1e006640 1 -- 172.21.15.79:0/292982959 shutdown_connections 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.393+0000 7fbd1e006640 1 -- 172.21.15.79:0/292982959 wait complete. 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.394+0000 7fbd1e006640 10 monclient: build_initial_monmap 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.394+0000 7fbd1e006640 10 monclient: monmap: 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:59.384 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 1 librados: starting msgr at 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 1 librados: starting objecter 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 1 Processor -- start 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 1 -- start start 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 1 librados: setting wanted keys 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 1 librados: calling monclient init 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.395+0000 7fbd1e006640 10 monclient: init 2024-03-29T10:47:59.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient(hunting): picked mon.noname-c con 0x7fbd180b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient(hunting): picked mon.noname-b con 0x7fbd181422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient(hunting): picked mon.noname-a con 0x7fbd18142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180abff0 con 0x7fbd18142690 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180ac260 con 0x7fbd181422b0 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18132a40 con 0x7fbd180b17d0 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd1e006640 10 monclient(hunting): authenticate will time out at 998.689725s 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd16ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fbd181422b0 0x7fbd18132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:58702/0 (socket says 172.21.15.79:58702) 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbd16ffd640 1 -- 172.21.15.79:0/1027934750 learned_addr learned my addr 172.21.15.79:0/1027934750 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:47:59.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.397+0000 7fbcfffff640 10 client.?.objecter ms_handle_connect 0x7fbd181422b0 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 client.?.objecter ms_handle_connect 0x7fbd18142690 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 client.?.objecter ms_handle_connect 0x7fbd180b17d0 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038903942 0 0) 0x7fbd180ac260 con 0x7fbd181422b0 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): my global_id is 4222 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd181422b0 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232717829 0 0) 0x7fbd180abff0 con 0x7fbd18142690 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.398+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180ac260 con 0x7fbd18142690 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815406186 0 0) 0x7fbd18132a40 con 0x7fbd180b17d0 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): my global_id is 4229 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd180abff0 con 0x7fbd180b17d0 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946912555 0 0) 0x7fbd04004eb0 con 0x7fbd181422b0 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd18132a40 con 0x7fbd181422b0 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbd0c001bc0 con 0x7fbd181422b0 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): dump: 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:59.388 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1947601713 0 0) 0x7fbd18132a40 con 0x7fbd181422b0 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 >> v1:172.21.15.79:6790/0 conn(0x7fbd180b17d0 legacy=0x7fbd181315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 >> v1:172.21.15.79:6789/0 conn(0x7fbd18142690 legacy=0x7fbd181e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 monclient: found mon.b 2024-03-29T10:47:59.389 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1812fdc0 con 0x7fbd181422b0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient: _finish_auth 0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbcfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbd1e006640 5 monclient: authenticate success, global_id 4222 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbd1e006640 10 monclient: _renew_subs 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbd1e006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.399+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18130070 con 0x7fbd181422b0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbd1e006640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbd1e006640 10 monclient: _renew_subs 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbd1e006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd180b71c0 con 0x7fbd181422b0 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbd1e006640 1 librados: init done 2024-03-29T10:47:59.390 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd0c001270 con 0x7fbd181422b0 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbcfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbd0c003a40 con 0x7fbd181422b0 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbcfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbcfffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbcfffff640 10 monclient: dump: 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:47:59.391 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:47:59.392 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:47:59.392 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:47:59.392 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:47:59.392 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.400+0000 7fbd157fa640 4 set_mon_vals no callback set 2024-03-29T10:47:59.393 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.402+0000 7fbd1e006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:59.393 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.402+0000 7fbd1e006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:59.393 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.402+0000 7fbd1e006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:59.393 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.402+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf8005080 con 0x7fbd181422b0 2024-03-29T10:47:59.393 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.403+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fbd0c003ba0 con 0x7fbd181422b0 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbd177fe640 10 monclient: get_auth_request con 0x7fbd0406de30 auth_method 0 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fbd0c084b20 con 0x7fbd181422b0 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbcfffff640 10 client.4222.objecter ms_dispatch 0x7fbd18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbcfffff640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbcfffff640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbcfffff640 20 client.4222.objecter dump_active .. 0 homeless 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.404+0000 7fbcfffff640 10 client.4222.objecter ms_handle_connect 0x7fbd0406de30 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.410+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd0c050400 con 0x7fbd181422b0 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.410+0000 7fbcfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:59.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.410+0000 7fbcfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:59.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.780+0000 7fbd1e006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.780+0000 7fbd1e006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.780+0000 7fbd1e006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:47:59.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.780+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf8005330 con 0x7fbd181422b0 2024-03-29T10:47:59.763 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.781+0000 7fbcfffff640 1 -- 172.21.15.79:0/1027934750 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fbd0c052330 con 0x7fbd181422b0 2024-03-29T10:47:59.763 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.781+0000 7fbcfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.763 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.781+0000 7fbcfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:59.763 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:47:59.763 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:59.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.784+0000 7fbd1e006640 10 librados: watch_flush enter 2024-03-29T10:47:59.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 10 librados: watch_flush exit 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 20 client.4222.objecter shutdown clearing up homeless session... 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 10 client.4222.objecter successfully canceled tick 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 >> v1:172.21.15.79:6816/4290694627 conn(0x7fbd0406de30 legacy=0x7fbd040702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 10 monclient: shutdown 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 >> v1:172.21.15.82:6789/0 conn(0x7fbd181422b0 legacy=0x7fbd18132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 shutdown_connections 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 >> 172.21.15.79:0/1027934750 conn(0x7fbd18066a00 msgr2=0x7fbd180b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 shutdown_connections 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 -- 172.21.15.79:0/1027934750 wait complete. 2024-03-29T10:47:59.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:47:59.785+0000 7fbd1e006640 1 librados: shutdown 2024-03-29T10:47:59.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:06.590 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:06.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.740+0000 7f8c8ea14640 10 monclient: get_monmap_and_config 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.740+0000 7f8c8ea14640 10 monclient: build_initial_monmap 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.740+0000 7f8c8ea14640 10 monclient: monmap: 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:06.741602+0000 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:06.741602+0000 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:06.723 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:06.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.742+0000 7f8c8ea14640 1 Processor -- start 2024-03-29T10:48:06.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.742+0000 7f8c8ea14640 1 -- start start 2024-03-29T10:48:06.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.742+0000 7f8c8ea14640 10 monclient: init 2024-03-29T10:48:06.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:06.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient(hunting): picked mon.noname-a con 0x7f8c881457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient(hunting): picked mon.noname-c con 0x7f8c8813f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient(hunting): picked mon.noname-b con 0x7f8c880a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880aaef0 con 0x7f8c881457f0 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88145bd0 con 0x7f8c8813f650 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88145eb0 con 0x7f8c880a9d70 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c8ea14640 10 monclient(hunting): authenticate will time out at 1006.035670s 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c877fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f8c8813f650 0x7f8c880b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:47842/0 (socket says 172.21.15.79:47842) 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c877fe640 1 -- 172.21.15.79:0/1657943953 learned_addr learned my addr 172.21.15.79:0/1657943953 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994072179 0 0) 0x7f8c88145eb0 con 0x7f8c880a9d70 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c86ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c86ffd640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:48:06.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.743+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c70003520 con 0x7f8c880a9d70 2024-03-29T10:48:06.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.744+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513185944 0 0) 0x7f8c70003520 con 0x7f8c880a9d70 2024-03-29T10:48:06.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.744+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 >> v1:172.21.15.82:6789/0 conn(0x7f8c8813f650 legacy=0x7f8c880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.744+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 >> v1:172.21.15.79:6789/0 conn(0x7f8c881457f0 legacy=0x7f8c8813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:06.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:06.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 1 monclient: found mon.noname-b 2024-03-29T10:48:06.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:06.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:48:06.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c88146340 con 0x7f8c880a9d70 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8c80001a10 con 0x7f8c880a9d70 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: dump: 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:06.729 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c8ea14640 5 monclient: authenticate success, global_id 4232 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c8ea14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c80001ca0 con 0x7f8c880a9d70 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c8ea14640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 1 -- 172.21.15.79:0/1657943953 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8c800020b0 con 0x7f8c880a9d70 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c8ea14640 4 set_mon_vals no callback set 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.746+0000 7f8c86ffd640 10 monclient: dump: 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:06.730 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:06.731 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 10 monclient: shutdown 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 1 -- 172.21.15.79:0/1657943953 >> v1:172.21.15.79:6790/0 conn(0x7f8c880a9d70 legacy=0x7f8c8814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 1 -- 172.21.15.79:0/1657943953 shutdown_connections 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 1 -- 172.21.15.79:0/1657943953 >> 172.21.15.79:0/1657943953 conn(0x7f8c88066a00 msgr2=0x7f8c88066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 1 -- 172.21.15.79:0/1657943953 shutdown_connections 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.747+0000 7f8c8ea14640 1 -- 172.21.15.79:0/1657943953 wait complete. 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.748+0000 7f8c8ea14640 10 monclient: build_initial_monmap 2024-03-29T10:48:06.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.748+0000 7f8c8ea14640 10 monclient: monmap: 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 1 librados: starting msgr at 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 1 librados: starting objecter 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 1 Processor -- start 2024-03-29T10:48:06.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 1 -- start start 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 1 librados: setting wanted keys 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 1 librados: calling monclient init 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.749+0000 7f8c8ea14640 10 monclient: init 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient(hunting): picked mon.noname-b con 0x7f8c880a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient(hunting): picked mon.noname-c con 0x7f8c8813f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient(hunting): picked mon.noname-a con 0x7f8c881457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88070860 con 0x7f8c881457f0 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c88071400 con 0x7f8c880a9d70 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880b17a0 con 0x7f8c8813f650 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c8ea14640 10 monclient(hunting): authenticate will time out at 1006.042590s 2024-03-29T10:48:06.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c87fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f8c880a9d70 0x7f8c8806ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:47952/0 (socket says 172.21.15.79:47952) 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c87fff640 1 -- 172.21.15.79:0/2741609449 learned_addr learned my addr 172.21.15.79:0/2741609449 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c84ff9640 10 client.?.objecter ms_handle_connect 0x7f8c8813f650 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c84ff9640 10 client.?.objecter ms_handle_connect 0x7f8c880a9d70 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.750+0000 7f8c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411195030 0 0) 0x7f8c880b17a0 con 0x7f8c8813f650 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c74004eb0 con 0x7f8c8813f650 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499072573 0 0) 0x7f8c88071400 con 0x7f8c880a9d70 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): my global_id is 4228 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c880b17a0 con 0x7f8c880a9d70 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345708027 0 0) 0x7f8c74004eb0 con 0x7f8c8813f650 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c88071400 con 0x7f8c8813f650 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8c780018a0 con 0x7f8c8813f650 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:06.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:06.744 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:06.744 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:06.744 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.744 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.744 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.744 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114844430 0 0) 0x7f8c880b17a0 con 0x7f8c880a9d70 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c74004eb0 con 0x7f8c880a9d70 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 473758884 0 0) 0x7f8c88071400 con 0x7f8c8813f650 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 >> v1:172.21.15.82:6789/0 conn(0x7f8c880a9d70 legacy=0x7f8c8806ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.751+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 >> v1:172.21.15.79:6789/0 conn(0x7f8c881457f0 legacy=0x7f8c8813a6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 1 monclient: found mon.c 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:06.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c881deab0 con 0x7f8c8813f650 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 5 monclient: authenticate success, global_id 4235 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 10 monclient: _renew_subs 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c881ded60 con 0x7f8c8813f650 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 10 monclient: _renew_subs 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:06.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8813afa0 con 0x7f8c8813f650 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c8ea14640 1 librados: init done 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c780047e0 con 0x7f8c8813f650 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8c78004b70 con 0x7f8c8813f650 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c84ff9640 10 monclient: dump: 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:06.747 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.753+0000 7f8c867fc640 4 set_mon_vals no callback set 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.755+0000 7f8c8ea14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.755+0000 7f8c8ea14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.755+0000 7f8c8ea14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.755+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c88075f10 con 0x7f8c8813f650 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.755+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f8c78004cd0 con 0x7f8c8813f650 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.756+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f8c78084a80 con 0x7f8c8813f650 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.756+0000 7f8c84ff9640 10 client.4235.objecter ms_dispatch 0x7f8c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.756+0000 7f8c84ff9640 3 client.4235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.756+0000 7f8c84ff9640 3 client.4235.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:06.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.756+0000 7f8c84ff9640 20 client.4235.objecter dump_active .. 0 homeless 2024-03-29T10:48:06.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.757+0000 7f8c87fff640 10 monclient: get_auth_request con 0x7f8c7406df60 auth_method 0 2024-03-29T10:48:06.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.757+0000 7f8c84ff9640 10 client.4235.objecter ms_handle_connect 0x7f8c7406df60 2024-03-29T10:48:06.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.764+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c78050360 con 0x7f8c8813f650 2024-03-29T10:48:06.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.764+0000 7f8c84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:06.764+0000 7f8c84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:07.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.066+0000 7f8c8ea14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.066+0000 7f8c8ea14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.066+0000 7f8c8ea14640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.066+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c880b3680 con 0x7f8c8813f650 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.067+0000 7f8c84ff9640 1 -- 172.21.15.79:0/2741609449 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f8c78052290 con 0x7f8c8813f650 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.067+0000 7f8c84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:07.050 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:07.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.067+0000 7f8c84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:07.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 10 librados: watch_flush enter 2024-03-29T10:48:07.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 10 librados: watch_flush exit 2024-03-29T10:48:07.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 20 client.4235.objecter shutdown clearing up homeless session... 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 10 client.4235.objecter successfully canceled tick 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 >> v1:172.21.15.79:6816/4290694627 conn(0x7f8c7406df60 legacy=0x7f8c74070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 10 monclient: shutdown 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.070+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 >> v1:172.21.15.79:6790/0 conn(0x7f8c8813f650 legacy=0x7f8c88070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.071+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 shutdown_connections 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.071+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 >> 172.21.15.79:0/2741609449 conn(0x7f8c88066a00 msgr2=0x7f8c8813dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.071+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 shutdown_connections 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.071+0000 7f8c8ea14640 1 -- 172.21.15.79:0/2741609449 wait complete. 2024-03-29T10:48:07.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:07.071+0000 7f8c8ea14640 1 librados: shutdown 2024-03-29T10:48:07.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:13.876 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.025+0000 7f93dbd48640 10 monclient: get_monmap_and_config 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.025+0000 7f93dbd48640 10 monclient: build_initial_monmap 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.025+0000 7f93dbd48640 10 monclient: monmap: 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:14.025940+0000 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:14.025940+0000 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:14.011 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 1 Processor -- start 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 1 -- start start 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient: init 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient(hunting): picked mon.noname-a con 0x7f93d41457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient(hunting): picked mon.noname-c con 0x7f93d413f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient(hunting): picked mon.noname-b con 0x7f93d40a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d40aaef0 con 0x7f93d41457f0 2024-03-29T10:48:14.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d4145bd0 con 0x7f93d413f650 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d4145eb0 con 0x7f93d40a9d70 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.026+0000 7f93dbd48640 10 monclient(hunting): authenticate will time out at 1013.319046s 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93da2e5640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f93d40a9d70 0x7f93d414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:59790/0 (socket says 172.21.15.79:59790) 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93da2e5640 1 -- 172.21.15.79:0/1760397324 learned_addr learned my addr 172.21.15.79:0/1760397324 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068572574 0 0) 0x7f93d4145eb0 con 0x7f93d40a9d70 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 10 monclient(hunting): _negotiate 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c0003520 con 0x7f93d40a9d70 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32303046 0 0) 0x7f93c0003520 con 0x7f93d40a9d70 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.027+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 >> v1:172.21.15.82:6789/0 conn(0x7f93d413f650 legacy=0x7f93d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:14.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.028+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 >> v1:172.21.15.79:6789/0 conn(0x7f93d41457f0 legacy=0x7f93d413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: _finish_hunting 0 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 1 monclient: found mon.noname-b 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4146340 con 0x7f93d40a9d70 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: _finish_auth 0 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: _check_auth_tickets 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93dbd48640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f93cc001a10 con 0x7f93d40a9d70 2024-03-29T10:48:14.017 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93dbd48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: dump: 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:14.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93dbd48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93cc001be0 con 0x7f93d40a9d70 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 1 -- 172.21.15.79:0/1760397324 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f93cc002030 con 0x7f93d40a9d70 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.030+0000 7f93d8ae2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93d8ae2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93d8ae2640 10 monclient: dump: 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:14.019 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 4 set_mon_vals no callback set 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 10 monclient: shutdown 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 1 -- 172.21.15.79:0/1760397324 >> v1:172.21.15.79:6790/0 conn(0x7f93d40a9d70 legacy=0x7f93d414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 1 -- 172.21.15.79:0/1760397324 shutdown_connections 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.031+0000 7f93dbd48640 1 -- 172.21.15.79:0/1760397324 >> 172.21.15.79:0/1760397324 conn(0x7f93d4066a00 msgr2=0x7f93d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.032+0000 7f93dbd48640 1 -- 172.21.15.79:0/1760397324 shutdown_connections 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.032+0000 7f93dbd48640 1 -- 172.21.15.79:0/1760397324 wait complete. 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 10 monclient: build_initial_monmap 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 10 monclient: monmap: 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:14.020 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 1 librados: starting msgr at 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 1 librados: starting objecter 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 1 Processor -- start 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 1 -- start start 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 1 librados: setting wanted keys 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 1 librados: calling monclient init 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.033+0000 7f93dbd48640 10 monclient: init 2024-03-29T10:48:14.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient(hunting): picked mon.noname-a con 0x7f93d40a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient(hunting): picked mon.noname-c con 0x7f93d413f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient(hunting): picked mon.noname-b con 0x7f93d41457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d4134300 con 0x7f93d40a9d70 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d4133730 con 0x7f93d41457f0 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d41338c0 con 0x7f93d413f650 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93dbd48640 10 monclient(hunting): authenticate will time out at 1013.326613s 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93d92e3640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f93d413f650 0x7f93d41d7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:33348/0 (socket says 172.21.15.79:33348) 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.034+0000 7f93d92e3640 1 -- 172.21.15.79:0/3465651613 learned_addr learned my addr 172.21.15.79:0/3465651613 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 client.?.objecter ms_handle_connect 0x7f93d413f650 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858514395 0 0) 0x7f93d41338c0 con 0x7f93d413f650 2024-03-29T10:48:14.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:48:14.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b8004eb0 con 0x7f93d413f650 2024-03-29T10:48:14.025 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352863974 0 0) 0x7f93b8004eb0 con 0x7f93d413f650 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93b8005300 con 0x7f93d413f650 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f93c4001920 con 0x7f93d413f650 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): dump: 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.035+0000 7f93ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.036+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 241733195 0 0) 0x7f93b8005300 con 0x7f93d413f650 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.036+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 >> v1:172.21.15.82:6789/0 conn(0x7f93d41457f0 legacy=0x7f93d41e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.036+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 >> v1:172.21.15.79:6789/0 conn(0x7f93d40a9d70 legacy=0x7f93d41361b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 1 monclient: found mon.c 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4133a50 con 0x7f93d413f650 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 5 monclient: authenticate success, global_id 4241 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 10 monclient: _renew_subs 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d41424c0 con 0x7f93d413f650 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 10 monclient: _renew_subs 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d4134790 con 0x7f93d413f650 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.037+0000 7f93dbd48640 1 librados: init done 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93dbd48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93dbd48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93dbd48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93c4001c40 con 0x7f93d413f650 2024-03-29T10:48:14.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4075f10 con 0x7f93d413f650 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f93c4001920 con 0x7f93d413f650 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.038+0000 7f93ca7fc640 10 monclient: dump: 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:14.029 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.039+0000 7f93cb7fe640 4 set_mon_vals no callback set 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.039+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f93c4003920 con 0x7f93d413f650 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f93c4084830 con 0x7f93d413f650 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93ca7fc640 10 client.4241.objecter ms_dispatch 0x7f93d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93ca7fc640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93ca7fc640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93ca7fc640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93d9ae4640 10 monclient: get_auth_request con 0x7f93b806e080 auth_method 0 2024-03-29T10:48:14.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.040+0000 7f93ca7fc640 10 client.4241.objecter ms_handle_connect 0x7f93b806e080 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.047+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93c4050110 con 0x7f93d413f650 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.047+0000 7f93ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.047+0000 7f93ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:14.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.347+0000 7f93dbd48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.347+0000 7f93dbd48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.347+0000 7f93dbd48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:14.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.347+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93d40b3680 con 0x7f93d413f650 2024-03-29T10:48:14.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.348+0000 7f93ca7fc640 1 -- 172.21.15.79:0/3465651613 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f93c4052040 con 0x7f93d413f650 2024-03-29T10:48:14.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.348+0000 7f93ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.348+0000 7f93ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:14.330 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:14.330 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 10 librados: watch_flush enter 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 10 librados: watch_flush exit 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 10 client.4241.objecter successfully canceled tick 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 >> v1:172.21.15.79:6816/4290694627 conn(0x7f93b806e080 legacy=0x7f93b8070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 10 monclient: shutdown 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 >> v1:172.21.15.79:6790/0 conn(0x7f93d413f650 legacy=0x7f93d41d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 shutdown_connections 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 >> 172.21.15.79:0/3465651613 conn(0x7f93d4066a00 msgr2=0x7f93d40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:14.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 shutdown_connections 2024-03-29T10:48:14.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 -- 172.21.15.79:0/3465651613 wait complete. 2024-03-29T10:48:14.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:14.350+0000 7f93dbd48640 1 librados: shutdown 2024-03-29T10:48:14.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:21.157 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:21.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.327+0000 7f16c102a640 10 monclient: get_monmap_and_config 2024-03-29T10:48:21.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.327+0000 7f16c102a640 10 monclient: build_initial_monmap 2024-03-29T10:48:21.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.327+0000 7f16c102a640 10 monclient: monmap: 2024-03-29T10:48:21.309 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:21.328159+0000 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:21.328159+0000 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:21.310 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:21.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.328+0000 7f16c102a640 1 Processor -- start 2024-03-29T10:48:21.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.328+0000 7f16c102a640 1 -- start start 2024-03-29T10:48:21.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.328+0000 7f16c102a640 10 monclient: init 2024-03-29T10:48:21.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient(hunting): picked mon.noname-a con 0x7f16bc0b0860 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient(hunting): picked mon.noname-c con 0x7f16bc072450 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient(hunting): picked mon.noname-b con 0x7f16bc13ee40 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0ad840 con 0x7f16bc0b0860 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc13c830 con 0x7f16bc072450 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc13c9c0 con 0x7f16bc13ee40 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:21.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.329+0000 7f16c102a640 10 monclient(hunting): authenticate will time out at 1020.621275s 2024-03-29T10:48:21.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16bb59e640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f16bc13ee40 0x7f16bc13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:52113/0 (socket says 172.21.15.79:52113) 2024-03-29T10:48:21.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16bb59e640 1 -- 172.21.15.79:0/597273107 learned_addr learned my addr 172.21.15.79:0/597273107 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435151441 0 0) 0x7f16bc13c9c0 con 0x7f16bc13ee40 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16b9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16b9d9b640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.333+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003520 con 0x7f16bc13ee40 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679304904 0 0) 0x7f16bc0ad840 con 0x7f16bc0b0860 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16bc13c9c0 con 0x7f16bc0b0860 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163501086 0 0) 0x7f16bc13c9c0 con 0x7f16bc0b0860 2024-03-29T10:48:21.319 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 >> v1:172.21.15.79:6790/0 conn(0x7f16bc13ee40 legacy=0x7f16bc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 >> v1:172.21.15.82:6789/0 conn(0x7f16bc072450 legacy=0x7f16bc13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 monclient: found mon.noname-a 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc13f970 con 0x7f16bc0b0860 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f16b4001920 con 0x7f16bc0b0860 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.320 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16b9d9b640 10 monclient: dump: 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16c102a640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.334+0000 7f16c102a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:21.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16b9d9b640 1 -- 172.21.15.79:0/597273107 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b4001b00 con 0x7f16bc0b0860 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16b9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16c102a640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16c102a640 4 set_mon_vals no callback set 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16c102a640 10 monclient: shutdown 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16c102a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.335+0000 7f16c102a640 1 -- 172.21.15.79:0/597273107 >> v1:172.21.15.79:6789/0 conn(0x7f16bc0b0860 legacy=0x7f16bc0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.336+0000 7f16c102a640 1 -- 172.21.15.79:0/597273107 shutdown_connections 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.336+0000 7f16c102a640 1 -- 172.21.15.79:0/597273107 >> 172.21.15.79:0/597273107 conn(0x7f16bc06b660 msgr2=0x7f16bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.336+0000 7f16c102a640 1 -- 172.21.15.79:0/597273107 shutdown_connections 2024-03-29T10:48:21.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.336+0000 7f16c102a640 1 -- 172.21.15.79:0/597273107 wait complete. 2024-03-29T10:48:21.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.337+0000 7f16c102a640 10 monclient: build_initial_monmap 2024-03-29T10:48:21.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.337+0000 7f16c102a640 10 monclient: monmap: 2024-03-29T10:48:21.323 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:21.323 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:21.323 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.337+0000 7f16c102a640 1 librados: starting msgr at 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.337+0000 7f16c102a640 1 librados: starting objecter 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.338+0000 7f16c102a640 1 Processor -- start 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.339+0000 7f16c102a640 1 -- start start 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.339+0000 7f16c102a640 1 librados: setting wanted keys 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.339+0000 7f16c102a640 1 librados: calling monclient init 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.339+0000 7f16c102a640 10 monclient: init 2024-03-29T10:48:21.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient(hunting): picked mon.noname-a con 0x7f16bc072450 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient(hunting): picked mon.noname-c con 0x7f16bc0b0860 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient(hunting): picked mon.noname-b con 0x7f16bc13ee40 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc0b2220 con 0x7f16bc072450 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc133e90 con 0x7f16bc13ee40 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc134930 con 0x7f16bc0b0860 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:21.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.340+0000 7f16c102a640 10 monclient(hunting): authenticate will time out at 1020.632449s 2024-03-29T10:48:21.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.345+0000 7f16bb59e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f16bc13ee40 0x7f16bc1dd510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:38119/0 (socket says 172.21.15.79:38119) 2024-03-29T10:48:21.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.345+0000 7f16bb59e640 1 -- 172.21.15.79:0/2729449282 learned_addr learned my addr 172.21.15.79:0/2729449282 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:21.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 client.?.objecter ms_handle_connect 0x7f16bc13ee40 2024-03-29T10:48:21.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 client.?.objecter ms_handle_connect 0x7f16bc072450 2024-03-29T10:48:21.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734474756 0 0) 0x7f16bc0b2220 con 0x7f16bc072450 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a8004eb0 con 0x7f16bc072450 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 client.?.objecter ms_handle_connect 0x7f16bc0b0860 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.346+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547658498 0 0) 0x7f16a8004eb0 con 0x7f16bc072450 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16bc0b2220 con 0x7f16bc072450 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894038723 0 0) 0x7f16bc134930 con 0x7f16bc0b0860 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:48:21.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a8004eb0 con 0x7f16bc0b0860 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f16b4003b10 con 0x7f16bc072450 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): dump: 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:21.333 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 979490473 0 0) 0x7f16bc0b2220 con 0x7f16bc072450 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 >> v1:172.21.15.79:6790/0 conn(0x7f16bc0b0860 legacy=0x7f16bc1d2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.347+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 >> v1:172.21.15.82:6789/0 conn(0x7f16bc13ee40 legacy=0x7f16bc1dd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 1 monclient: found mon.a 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc134ba0 con 0x7f16bc072450 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16c102a640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:48:21.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16c102a640 10 monclient: _renew_subs 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b4003f20 con 0x7f16bc072450 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16c102a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc132890 con 0x7f16bc072450 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f16b40042f0 con 0x7f16bc072450 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.348+0000 7f16a37fe640 10 monclient: dump: 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.335 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:21.336 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:21.336 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:21.336 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:21.336 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.349+0000 7f16c102a640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.349+0000 7f16c102a640 10 monclient: _renew_subs 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.349+0000 7f16c102a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.349+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc1d36f0 con 0x7f16bc072450 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.349+0000 7f16c102a640 1 librados: init done 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.349+0000 7f16b959a640 4 set_mon_vals no callback set 2024-03-29T10:48:21.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.352+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f16b4001db0 con 0x7f16bc072450 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.352+0000 7f16c102a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.352+0000 7f16c102a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.352+0000 7f16c102a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.352+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c005080 con 0x7f16bc072450 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16ba59c640 10 monclient: get_auth_request con 0x7f16a806db40 auth_method 0 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f16b4084d90 con 0x7f16bc072450 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16a37fe640 10 client.4251.objecter ms_dispatch 0x7f16bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16a37fe640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16a37fe640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16a37fe640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T10:48:21.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.353+0000 7f16a37fe640 10 client.4251.objecter ms_handle_connect 0x7f16a806db40 2024-03-29T10:48:21.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.360+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16b404f640 con 0x7f16bc072450 2024-03-29T10:48:21.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.360+0000 7f16a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.360+0000 7f16a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:21.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.684+0000 7f16c102a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.684+0000 7f16c102a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.684+0000 7f16c102a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:21.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.684+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f169c005330 con 0x7f16bc072450 2024-03-29T10:48:21.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.690+0000 7f16a37fe640 1 -- 172.21.15.79:0/2729449282 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f16b4051570 con 0x7f16bc072450 2024-03-29T10:48:21.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.690+0000 7f16a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.690+0000 7f16a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:21.673 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:21.673 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:21.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.694+0000 7f16c102a640 10 librados: watch_flush enter 2024-03-29T10:48:21.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 10 librados: watch_flush exit 2024-03-29T10:48:21.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T10:48:21.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 10 client.4251.objecter successfully canceled tick 2024-03-29T10:48:21.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 >> v1:172.21.15.79:6816/4290694627 conn(0x7f16a806db40 legacy=0x7f16a8070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 10 monclient: shutdown 2024-03-29T10:48:21.680 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:21.680 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.696+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 >> v1:172.21.15.79:6789/0 conn(0x7f16bc072450 legacy=0x7f16bc133530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.680 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.697+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 shutdown_connections 2024-03-29T10:48:21.680 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.697+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 >> 172.21.15.79:0/2729449282 conn(0x7f16bc06b660 msgr2=0x7f16bc13bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:21.682 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.698+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 shutdown_connections 2024-03-29T10:48:21.682 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.698+0000 7f16c102a640 1 -- 172.21.15.79:0/2729449282 wait complete. 2024-03-29T10:48:21.682 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:21.700+0000 7f16c102a640 1 librados: shutdown 2024-03-29T10:48:21.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:28.507 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.679+0000 7fc9ad57a640 10 monclient: get_monmap_and_config 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.679+0000 7fc9ad57a640 10 monclient: build_initial_monmap 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.679+0000 7fc9ad57a640 10 monclient: monmap: 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:28.681656+0000 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:28.681656+0000 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:28.663 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:28.664 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:28.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.681+0000 7fc9ad57a640 1 Processor -- start 2024-03-29T10:48:28.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.682+0000 7fc9ad57a640 1 -- start start 2024-03-29T10:48:28.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.682+0000 7fc9ad57a640 10 monclient: init 2024-03-29T10:48:28.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a81457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a813f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a80a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80aaef0 con 0x7fc9a813f650 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a8145bd0 con 0x7fc9a80a9d70 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a8145eb0 con 0x7fc9a81457f0 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:28.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.683+0000 7fc9ad57a640 10 monclient(hunting): authenticate will time out at 1027.976311s 2024-03-29T10:48:28.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a77fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fc9a80a9d70 0x7fc9a814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:52523/0 (socket says 172.21.15.79:52523) 2024-03-29T10:48:28.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a77fe640 1 -- 172.21.15.79:0/2711210204 learned_addr learned my addr 172.21.15.79:0/2711210204 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:28.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184045232 0 0) 0x7fc9a8145bd0 con 0x7fc9a80a9d70 2024-03-29T10:48:28.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a5ffb640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.686+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc990003520 con 0x7fc9a80a9d70 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.687+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299051364 0 0) 0x7fc9a80aaef0 con 0x7fc9a813f650 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.687+0000 7fc9a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.687+0000 7fc9a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.687+0000 7fc9a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.687+0000 7fc9a5ffb640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.687+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a8145bd0 con 0x7fc9a813f650 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1872708849 0 0) 0x7fc9a8145bd0 con 0x7fc9a813f650 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 >> v1:172.21.15.79:6790/0 conn(0x7fc9a81457f0 legacy=0x7fc9a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 >> v1:172.21.15.82:6789/0 conn(0x7fc9a80a9d70 legacy=0x7fc9a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 1 monclient: found mon.noname-a 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a8146340 con 0x7fc9a813f650 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc99c001670 con 0x7fc9a813f650 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.688+0000 7fc9a5ffb640 10 monclient: dump: 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.675 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc99c0017d0 con 0x7fc9a813f650 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9a5ffb640 1 -- 172.21.15.79:0/2711210204 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc99c001be0 con 0x7fc9a813f650 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9a5ffb640 10 monclient: dump: 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:28.676 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9ad57a640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9ad57a640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:28.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.689+0000 7fc9ad57a640 4 set_mon_vals no callback set 2024-03-29T10:48:28.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.690+0000 7fc9ad57a640 10 monclient: shutdown 2024-03-29T10:48:28.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.690+0000 7fc9ad57a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:28.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.690+0000 7fc9ad57a640 1 -- 172.21.15.79:0/2711210204 >> v1:172.21.15.79:6789/0 conn(0x7fc9a813f650 legacy=0x7fc9a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.690+0000 7fc9ad57a640 1 -- 172.21.15.79:0/2711210204 shutdown_connections 2024-03-29T10:48:28.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.690+0000 7fc9ad57a640 1 -- 172.21.15.79:0/2711210204 >> 172.21.15.79:0/2711210204 conn(0x7fc9a8066a00 msgr2=0x7fc9a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:28.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.692+0000 7fc9ad57a640 1 -- 172.21.15.79:0/2711210204 shutdown_connections 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.692+0000 7fc9ad57a640 1 -- 172.21.15.79:0/2711210204 wait complete. 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.693+0000 7fc9ad57a640 10 monclient: build_initial_monmap 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.693+0000 7fc9ad57a640 10 monclient: monmap: 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 1 librados: starting msgr at 2024-03-29T10:48:28.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 1 librados: starting objecter 2024-03-29T10:48:28.680 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 1 Processor -- start 2024-03-29T10:48:28.683 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 1 -- start start 2024-03-29T10:48:28.683 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 1 librados: setting wanted keys 2024-03-29T10:48:28.683 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 1 librados: calling monclient init 2024-03-29T10:48:28.683 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.694+0000 7fc9ad57a640 10 monclient: init 2024-03-29T10:48:28.683 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:28.683 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a80a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a81457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a8141e80 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a80719b0 con 0x7fc9a8141e80 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a81dbf60 con 0x7fc9a81457f0 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a81dbc80 con 0x7fc9a80a9d70 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.695+0000 7fc9ad57a640 10 monclient(hunting): authenticate will time out at 1027.988551s 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc9a67fc640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fc9a81457f0 0x7fc9a806f2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:52579/0 (socket says 172.21.15.79:52579) 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc9a67fc640 1 -- 172.21.15.79:0/3049906146 learned_addr learned my addr 172.21.15.79:0/3049906146 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 10 client.?.objecter ms_handle_connect 0x7fc9a81457f0 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414209020 0 0) 0x7fc9a81dbf60 con 0x7fc9a81457f0 2024-03-29T10:48:28.684 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.698+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc994004eb0 con 0x7fc9a81457f0 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 150964097 0 0) 0x7fc994004eb0 con 0x7fc9a81457f0 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9a81dbf60 con 0x7fc9a81457f0 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc99c001be0 con 0x7fc9a81457f0 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 10 monclient(hunting): dump: 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.685 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.699+0000 7fc98f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.700+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3438612356 0 0) 0x7fc9a81dbf60 con 0x7fc9a81457f0 2024-03-29T10:48:28.686 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.700+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 >> v1:172.21.15.79:6790/0 conn(0x7fc9a80a9d70 legacy=0x7fc9a81db570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 >> v1:172.21.15.79:6789/0 conn(0x7fc9a8141e80 legacy=0x7fc9a81e5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 1 monclient: found mon.b 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a806f9f0 con 0x7fc9a81457f0 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc98f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:28.689 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 10 monclient: _renew_subs 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a806fc40 con 0x7fc9a81457f0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 10 monclient: _renew_subs 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81e78b0 con 0x7fc9a81457f0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.701+0000 7fc9ad57a640 1 librados: init done 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc9ad57a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc9ad57a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc9ad57a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a81dc150 con 0x7fc9a81457f0 2024-03-29T10:48:28.690 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc99c0038a0 con 0x7fc9a81457f0 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc98f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc99c003cf0 con 0x7fc9a81457f0 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc98f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc98f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc98f7fe640 10 monclient: dump: 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:28.691 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.703+0000 7fc9a57fa640 4 set_mon_vals no callback set 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.705+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fc99c003e50 con 0x7fc9a81457f0 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fc99c084bf0 con 0x7fc9a81457f0 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc98f7fe640 10 client.4234.objecter ms_dispatch 0x7fc9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc98f7fe640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc98f7fe640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc9a6ffd640 10 monclient: get_auth_request con 0x7fc99406dc90 auth_method 0 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc98f7fe640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T10:48:28.692 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.706+0000 7fc98f7fe640 10 client.4234.objecter ms_handle_connect 0x7fc99406dc90 2024-03-29T10:48:28.695 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.712+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc99c0504d0 con 0x7fc9a81457f0 2024-03-29T10:48:28.695 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.712+0000 7fc98f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.695 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:28.712+0000 7fc98f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.032+0000 7fc9ad57a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.032+0000 7fc9ad57a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.032+0000 7fc9ad57a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.032+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9a8075f10 con 0x7fc9a81457f0 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.033+0000 7fc98f7fe640 1 -- 172.21.15.79:0/3049906146 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fc99c052400 con 0x7fc9a81457f0 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.033+0000 7fc98f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:29.016 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.033+0000 7fc98f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:29.019 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:29.019 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:29.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.036+0000 7fc9ad57a640 10 librados: watch_flush enter 2024-03-29T10:48:29.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.036+0000 7fc9ad57a640 10 librados: watch_flush exit 2024-03-29T10:48:29.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.036+0000 7fc9ad57a640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T10:48:29.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.037+0000 7fc9ad57a640 10 client.4234.objecter successfully canceled tick 2024-03-29T10:48:29.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.037+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 >> v1:172.21.15.79:6816/4290694627 conn(0x7fc99406dc90 legacy=0x7fc994070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:29.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.037+0000 7fc9ad57a640 10 monclient: shutdown 2024-03-29T10:48:29.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.037+0000 7fc9ad57a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:29.021 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.037+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 >> v1:172.21.15.82:6789/0 conn(0x7fc9a81457f0 legacy=0x7fc9a806f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:29.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.040+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 shutdown_connections 2024-03-29T10:48:29.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.040+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 >> 172.21.15.79:0/3049906146 conn(0x7fc9a8066a00 msgr2=0x7fc9a8136300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:29.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.040+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 shutdown_connections 2024-03-29T10:48:29.024 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.040+0000 7fc9ad57a640 1 -- 172.21.15.79:0/3049906146 wait complete. 2024-03-29T10:48:29.024 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:29.041+0000 7fc9ad57a640 1 librados: shutdown 2024-03-29T10:48:29.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:35.849 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:35.998 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.016+0000 7fe56336d640 10 monclient: get_monmap_and_config 2024-03-29T10:48:35.998 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.016+0000 7fe56336d640 10 monclient: build_initial_monmap 2024-03-29T10:48:35.998 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.016+0000 7fe56336d640 10 monclient: monmap: 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:36.017466+0000 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:36.017466+0000 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:35.999 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.017+0000 7fe56336d640 1 Processor -- start 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.017+0000 7fe56336d640 1 -- start start 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.017+0000 7fe56336d640 10 monclient: init 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient(hunting): picked mon.noname-b con 0x7fe55c0b2800 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:36.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient(hunting): picked mon.noname-c con 0x7fe55c0b3220 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:36.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient(hunting): picked mon.noname-a con 0x7fe55c1439c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:36.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c0b3840 con 0x7fe55c1439c0 2024-03-29T10:48:36.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c06f230 con 0x7fe55c0b3220 2024-03-29T10:48:36.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c06ef50 con 0x7fe55c0b2800 2024-03-29T10:48:36.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:36.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.018+0000 7fe56336d640 10 monclient(hunting): authenticate will time out at 1035.310862s 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe56236b640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe55c0b2800 0x7fe55c1432b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43525/0 (socket says 172.21.15.79:43525) 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe56236b640 1 -- 172.21.15.79:0/403227527 learned_addr learned my addr 172.21.15.79:0/403227527 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662351475 0 0) 0x7fe55c06ef50 con 0x7fe55c0b2800 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe561369640 10 monclient(hunting): _negotiate 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe561369640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe561369640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe561369640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:48:36.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.026+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe548003520 con 0x7fe55c0b2800 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264542982 0 0) 0x7fe548003520 con 0x7fe55c0b2800 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 >> v1:172.21.15.82:6789/0 conn(0x7fe55c0b3220 legacy=0x7fe55c146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 >> v1:172.21.15.79:6789/0 conn(0x7fe55c1439c0 legacy=0x7fe55c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: _finish_hunting 0 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 1 monclient: found mon.noname-b 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c071d30 con 0x7fe55c0b2800 2024-03-29T10:48:36.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: _finish_auth 0 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: _check_auth_tickets 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe558001690 con 0x7fe55c0b2800 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe561369640 10 monclient: dump: 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:36.011 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:36.012 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:36.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe56336d640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:48:36.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.027+0000 7fe56336d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.031+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe558001860 con 0x7fe55c0b2800 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.031+0000 7fe561369640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.031+0000 7fe561369640 1 -- 172.21.15.79:0/403227527 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe558001cb0 con 0x7fe55c0b2800 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.031+0000 7fe561369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.031+0000 7fe561369640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.031+0000 7fe561369640 10 monclient: dump: 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.013 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.014 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:36.014 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:36.014 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:36.014 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:36.014 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:36.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.033+0000 7fe56336d640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:36.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.033+0000 7fe56336d640 4 set_mon_vals no callback set 2024-03-29T10:48:36.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.034+0000 7fe56336d640 10 monclient: shutdown 2024-03-29T10:48:36.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.034+0000 7fe56336d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.034+0000 7fe56336d640 1 -- 172.21.15.79:0/403227527 >> v1:172.21.15.79:6790/0 conn(0x7fe55c0b2800 legacy=0x7fe55c1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.034+0000 7fe56336d640 1 -- 172.21.15.79:0/403227527 shutdown_connections 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.034+0000 7fe56336d640 1 -- 172.21.15.79:0/403227527 >> 172.21.15.79:0/403227527 conn(0x7fe55c066a00 msgr2=0x7fe55c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.036+0000 7fe56336d640 1 -- 172.21.15.79:0/403227527 shutdown_connections 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.036+0000 7fe56336d640 1 -- 172.21.15.79:0/403227527 wait complete. 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.037+0000 7fe56336d640 10 monclient: build_initial_monmap 2024-03-29T10:48:36.019 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.037+0000 7fe56336d640 10 monclient: monmap: 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 librados: starting msgr at 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 librados: starting objecter 2024-03-29T10:48:36.020 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 Processor -- start 2024-03-29T10:48:36.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 -- start start 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 librados: setting wanted keys 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 librados: calling monclient init 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 10 monclient: init 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 10 monclient(hunting): picked mon.noname-b con 0x7fe55c0b2800 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 10 monclient(hunting): picked mon.noname-a con 0x7fe55c0b3220 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 10 monclient(hunting): picked mon.noname-c con 0x7fe55c1439c0 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.038+0000 7fe56336d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c149d10 con 0x7fe55c0b3220 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.039+0000 7fe56336d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c0aaa20 con 0x7fe55c0b2800 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.039+0000 7fe56336d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55c1e1600 con 0x7fe55c1439c0 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.039+0000 7fe56336d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:36.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.039+0000 7fe56336d640 10 monclient(hunting): authenticate will time out at 1035.331128s 2024-03-29T10:48:36.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.043+0000 7fe56236b640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe55c0b2800 0x7fe55c1e3980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:40139/0 (socket says 172.21.15.79:40139) 2024-03-29T10:48:36.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.043+0000 7fe56236b640 1 -- 172.21.15.79:0/4177995623 learned_addr learned my addr 172.21.15.79:0/4177995623 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:36.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.043+0000 7fe552ffd640 10 client.?.objecter ms_handle_connect 0x7fe55c0b2800 2024-03-29T10:48:36.026 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.043+0000 7fe552ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 client.?.objecter ms_handle_connect 0x7fe55c1439c0 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102270510 0 0) 0x7fe55c1e1600 con 0x7fe55c1439c0 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe540004eb0 con 0x7fe55c1439c0 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2791887217 0 0) 0x7fe540004eb0 con 0x7fe55c1439c0 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe540005300 con 0x7fe55c1439c0 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe54c001710 con 0x7fe55c1439c0 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): dump: 2024-03-29T10:48:36.027 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 106808215 0 0) 0x7fe540005300 con 0x7fe55c1439c0 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 >> v1:172.21.15.82:6789/0 conn(0x7fe55c0b2800 legacy=0x7fe55c1e3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.028 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.044+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 >> v1:172.21.15.79:6789/0 conn(0x7fe55c0b3220 legacy=0x7fe55c1e07e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 1 monclient: found mon.c 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c1499e0 con 0x7fe55c1439c0 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe54c001cc0 con 0x7fe55c1439c0 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe54c001710 con 0x7fe55c1439c0 2024-03-29T10:48:36.029 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 10 monclient: _renew_subs 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe560b68640 4 set_mon_vals no callback set 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55c0aac90 con 0x7fe55c1439c0 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe552ffd640 10 monclient: dump: 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:36.030 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 10 monclient: _renew_subs 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55c14a260 con 0x7fe55c1439c0 2024-03-29T10:48:36.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.045+0000 7fe56336d640 1 librados: init done 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.048+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fe54c019460 con 0x7fe55c1439c0 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe550ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe550ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe550ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55c04f730 con 0x7fe55c1439c0 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fe54c0832f0 con 0x7fe55c1439c0 2024-03-29T10:48:36.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe552ffd640 10 client.4253.objecter ms_dispatch 0x7fe55c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:36.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe552ffd640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:36.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe552ffd640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:36.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe552ffd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T10:48:36.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.049+0000 7fe56236b640 10 monclient: get_auth_request con 0x7fe54006e360 auth_method 0 2024-03-29T10:48:36.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.050+0000 7fe552ffd640 10 client.4253.objecter ms_handle_connect 0x7fe54006e360 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.060+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe54c04ea00 con 0x7fe55c1439c0 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.060+0000 7fe552ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:36.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.060+0000 7fe552ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:36.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.376+0000 7fe550ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.376+0000 7fe550ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.376+0000 7fe550ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:48:36.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.376+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe55c074d80 con 0x7fe55c1439c0 2024-03-29T10:48:36.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.377+0000 7fe552ffd640 1 -- 172.21.15.79:0/4177995623 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fe54c04ea00 con 0x7fe55c1439c0 2024-03-29T10:48:36.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.377+0000 7fe552ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.377+0000 7fe552ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:36.359 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:36.359 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.380+0000 7fe550ff9640 10 librados: watch_flush enter 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 10 librados: watch_flush exit 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 10 client.4253.objecter successfully canceled tick 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 >> v1:172.21.15.79:6816/4290694627 conn(0x7fe54006e360 legacy=0x7fe540070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 10 monclient: shutdown 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:36.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 >> v1:172.21.15.79:6790/0 conn(0x7fe55c1439c0 legacy=0x7fe55c1e0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 shutdown_connections 2024-03-29T10:48:36.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.381+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 >> 172.21.15.79:0/4177995623 conn(0x7fe55c066a00 msgr2=0x7fe55c141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:36.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.383+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 shutdown_connections 2024-03-29T10:48:36.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.383+0000 7fe550ff9640 1 -- 172.21.15.79:0/4177995623 wait complete. 2024-03-29T10:48:36.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:36.383+0000 7fe550ff9640 1 librados: shutdown 2024-03-29T10:48:36.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:43.190 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.358+0000 7f645e1f5640 10 monclient: get_monmap_and_config 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.358+0000 7f645e1f5640 10 monclient: build_initial_monmap 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.358+0000 7f645e1f5640 10 monclient: monmap: 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:43.359047+0000 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:43.359047+0000 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:43.341 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:43.342 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:43.342 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.359+0000 7f645e1f5640 1 Processor -- start 2024-03-29T10:48:43.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.359+0000 7f645e1f5640 1 -- start start 2024-03-29T10:48:43.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.359+0000 7f645e1f5640 10 monclient: init 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient(hunting): picked mon.noname-a con 0x7f6450006a70 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient(hunting): picked mon.noname-c con 0x7f645002b100 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient(hunting): picked mon.noname-b con 0x7f6450029c30 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645002a270 con 0x7f6450006a70 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645002c440 con 0x7f645002b100 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64500c9440 con 0x7f6450029c30 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:43.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.360+0000 7f645e1f5640 10 monclient(hunting): authenticate will time out at 1042.652729s 2024-03-29T10:48:43.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.364+0000 7f645d9f4640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f6450029c30 0x7f64500d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:33645/0 (socket says 172.21.15.79:33645) 2024-03-29T10:48:43.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.364+0000 7f645d9f4640 1 -- 172.21.15.79:0/1739712591 learned_addr learned my addr 172.21.15.79:0/1739712591 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:43.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.368+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278659986 0 0) 0x7f645002a270 con 0x7f6450006a70 2024-03-29T10:48:43.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.368+0000 7f644ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.368+0000 7f644ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.368+0000 7f644ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.368+0000 7f644ffff640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:48:43.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.368+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6444003520 con 0x7f6450006a70 2024-03-29T10:48:43.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227184420 0 0) 0x7f64500c9440 con 0x7f6450029c30 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645002a270 con 0x7f6450029c30 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224186791 0 0) 0x7f6444003520 con 0x7f6450006a70 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 >> v1:172.21.15.79:6790/0 conn(0x7f6450029c30 legacy=0x7f64500d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 >> v1:172.21.15.82:6789/0 conn(0x7f645002b100 legacy=0x7f64500cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 monclient: found mon.noname-a 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64500c95d0 con 0x7f6450006a70 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:43.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6454001610 con 0x7f6450006a70 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f645e1f5640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f645e1f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f644ffff640 10 monclient: dump: 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:43.357 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:43.358 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.372+0000 7f645e1f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:43.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.374+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64540017e0 con 0x7f6450006a70 2024-03-29T10:48:43.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.374+0000 7f644ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:43.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.375+0000 7f645e1f5640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:43.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.375+0000 7f645e1f5640 4 set_mon_vals no callback set 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.375+0000 7f644ffff640 1 -- 172.21.15.79:0/1739712591 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6454001c30 con 0x7f6450006a70 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.375+0000 7f644ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.375+0000 7f644ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.375+0000 7f644ffff640 10 monclient: dump: 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 10 monclient: shutdown 2024-03-29T10:48:43.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:43.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 1 -- 172.21.15.79:0/1739712591 >> v1:172.21.15.79:6789/0 conn(0x7f6450006a70 legacy=0x7f64500c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 1 -- 172.21.15.79:0/1739712591 shutdown_connections 2024-03-29T10:48:43.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 1 -- 172.21.15.79:0/1739712591 >> 172.21.15.79:0/1739712591 conn(0x7f64500058f0 msgr2=0x7f6450005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:43.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 1 -- 172.21.15.79:0/1739712591 shutdown_connections 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.376+0000 7f645e1f5640 1 -- 172.21.15.79:0/1739712591 wait complete. 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.377+0000 7f645e1f5640 10 monclient: build_initial_monmap 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.377+0000 7f645e1f5640 10 monclient: monmap: 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:43.363 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 1 librados: starting msgr at 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 1 librados: starting objecter 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 1 Processor -- start 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 1 -- start start 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 1 librados: setting wanted keys 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 1 librados: calling monclient init 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.378+0000 7f645e1f5640 10 monclient: init 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient(hunting): picked mon.noname-b con 0x7f6450029c30 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient(hunting): picked mon.noname-c con 0x7f645016a1d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:43.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient(hunting): picked mon.noname-a con 0x7f6450166de0 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:43.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645002edd0 con 0x7f6450166de0 2024-03-29T10:48:43.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645002f0b0 con 0x7f6450029c30 2024-03-29T10:48:43.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645002d0d0 con 0x7f645016a1d0 2024-03-29T10:48:43.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:43.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.379+0000 7f645e1f5640 10 monclient(hunting): authenticate will time out at 1042.671900s 2024-03-29T10:48:43.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.386+0000 7f645d9f4640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f6450166de0 0x7f645002dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:57053/0 (socket says 172.21.15.79:57053) 2024-03-29T10:48:43.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.386+0000 7f645d9f4640 1 -- 172.21.15.79:0/2300565848 learned_addr learned my addr 172.21.15.79:0/2300565848 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 10 client.?.objecter ms_handle_connect 0x7f6450166de0 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 10 client.?.objecter ms_handle_connect 0x7f645016a1d0 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 10 client.?.objecter ms_handle_connect 0x7f6450029c30 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.387+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710116104 0 0) 0x7f645002edd0 con 0x7f6450166de0 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:48:43.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c004eb0 con 0x7f6450166de0 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513311698 0 0) 0x7f645002d0d0 con 0x7f645016a1d0 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645002edd0 con 0x7f645016a1d0 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474893454 0 0) 0x7f645002f0b0 con 0x7f6450029c30 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:48:43.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.388+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f645002d0d0 con 0x7f6450029c30 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091622188 0 0) 0x7f645002d0d0 con 0x7f6450029c30 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f645002f0b0 con 0x7f6450029c30 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6454001af0 con 0x7f6450029c30 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 10 monclient(hunting): dump: 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.373 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.389+0000 7f644dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.390+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3444053409 0 0) 0x7f645002edd0 con 0x7f645016a1d0 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.390+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f645002d0d0 con 0x7f645016a1d0 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.390+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6448001750 con 0x7f645016a1d0 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.390+0000 7f644dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.390+0000 7f644dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.390+0000 7f644dffb640 10 monclient(hunting): dump: 2024-03-29T10:48:43.374 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:43.375 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2677758045 0 0) 0x7f645002f0b0 con 0x7f6450029c30 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 >> v1:172.21.15.79:6790/0 conn(0x7f645016a1d0 legacy=0x7f645002d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 >> v1:172.21.15.79:6789/0 conn(0x7f6450166de0 legacy=0x7f645002dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 1 monclient: found mon.b 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645002cd30 con 0x7f6450029c30 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:43.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64540010a0 con 0x7f6450029c30 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.392+0000 7f644f7fe640 4 set_mon_vals no callback set 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6454003a40 con 0x7f6450029c30 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f644dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f644dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f644dffb640 10 monclient: dump: 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:43.377 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 10 monclient: _renew_subs 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64501697b0 con 0x7f6450029c30 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 10 monclient: _renew_subs 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6450169b50 con 0x7f6450029c30 2024-03-29T10:48:43.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.393+0000 7f645e1f5640 1 librados: init done 2024-03-29T10:48:43.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.395+0000 7f645e1f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.395+0000 7f645e1f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.395+0000 7f645e1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.395+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6450004cd0 con 0x7f6450029c30 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.396+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f6454019a10 con 0x7f6450029c30 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.397+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f64540836b0 con 0x7f6450029c30 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.397+0000 7f644dffb640 10 client.4240.objecter ms_dispatch 0x7f6458000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.397+0000 7f644dffb640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.397+0000 7f644dffb640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.397+0000 7f645c9f2640 10 monclient: get_auth_request con 0x7f643c06db40 auth_method 0 2024-03-29T10:48:43.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.397+0000 7f644dffb640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T10:48:43.382 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.400+0000 7f644dffb640 10 client.4240.objecter ms_handle_connect 0x7f643c06db40 2024-03-29T10:48:43.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.405+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f645404ef90 con 0x7f6450029c30 2024-03-29T10:48:43.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.405+0000 7f644dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.405+0000 7f644dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:43.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.741+0000 7f645e1f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.741+0000 7f645e1f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.742+0000 7f645e1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:48:43.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.742+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6450006e50 con 0x7f6450029c30 2024-03-29T10:48:43.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.743+0000 7f644dffb640 1 -- 172.21.15.79:0/2300565848 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f6454050ec0 con 0x7f6450029c30 2024-03-29T10:48:43.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.744+0000 7f644dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.726 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:43.726 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:43.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.744+0000 7f644dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:43.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.747+0000 7f645e1f5640 10 librados: watch_flush enter 2024-03-29T10:48:43.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.747+0000 7f645e1f5640 10 librados: watch_flush exit 2024-03-29T10:48:43.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.747+0000 7f645e1f5640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T10:48:43.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.747+0000 7f645e1f5640 10 client.4240.objecter successfully canceled tick 2024-03-29T10:48:43.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.749+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 >> v1:172.21.15.79:6816/4290694627 conn(0x7f643c06db40 legacy=0x7f643c070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.749+0000 7f645e1f5640 10 monclient: shutdown 2024-03-29T10:48:43.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.749+0000 7f645e1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:43.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.749+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 >> v1:172.21.15.82:6789/0 conn(0x7f6450029c30 legacy=0x7f64501666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.749+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 shutdown_connections 2024-03-29T10:48:43.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.749+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 >> 172.21.15.79:0/2300565848 conn(0x7f64500058f0 msgr2=0x7f6450034e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:43.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.752+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 shutdown_connections 2024-03-29T10:48:43.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.753+0000 7f645e1f5640 1 -- 172.21.15.79:0/2300565848 wait complete. 2024-03-29T10:48:43.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:43.753+0000 7f645e1f5640 1 librados: shutdown 2024-03-29T10:48:43.755 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:50.562 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.731+0000 7f027e049640 10 monclient: get_monmap_and_config 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.731+0000 7f027e049640 10 monclient: build_initial_monmap 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.731+0000 7f027e049640 10 monclient: monmap: 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:50.732054+0000 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:50.732054+0000 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:50.713 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:50.714 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:50.714 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:50.714 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:50.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.732+0000 7f027e049640 1 Processor -- start 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.732+0000 7f027e049640 1 -- start start 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.732+0000 7f027e049640 10 monclient: init 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient(hunting): picked mon.noname-a con 0x7f02780b1f60 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient(hunting): picked mon.noname-b con 0x7f0278071eb0 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient(hunting): picked mon.noname-c con 0x7f0278143d10 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027806ef80 con 0x7f02780b1f60 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027806f260 con 0x7f0278143d10 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02780b40c0 con 0x7f0278071eb0 2024-03-29T10:48:50.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:50.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.733+0000 7f027e049640 10 monclient(hunting): authenticate will time out at 1050.025425s 2024-03-29T10:48:50.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.737+0000 7f02777fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f02780b1f60 0x7f0278143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:40899/0 (socket says 172.21.15.79:40899) 2024-03-29T10:48:50.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.737+0000 7f02777fe640 1 -- 172.21.15.79:0/3125580115 learned_addr learned my addr 172.21.15.79:0/3125580115 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:50.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929925434 0 0) 0x7f027806ef80 con 0x7f02780b1f60 2024-03-29T10:48:50.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0260003520 con 0x7f02780b1f60 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566116591 0 0) 0x7f0260003520 con 0x7f02780b1f60 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.738+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 >> v1:172.21.15.79:6790/0 conn(0x7f0278071eb0 legacy=0x7f0278146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:50.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.739+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 >> v1:172.21.15.82:6789/0 conn(0x7f0278143d10 legacy=0x7f0278146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:50.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:50.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 1 monclient: found mon.noname-a 2024-03-29T10:48:50.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:50.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:48:50.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0278155d70 con 0x7f02780b1f60 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0270001a10 con 0x7f02780b1f60 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: dump: 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:50.726 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0270001be0 con 0x7f02780b1f60 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 1 -- 172.21.15.79:0/3125580115 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0270002030 con 0x7f02780b1f60 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.742+0000 7f02767fc640 10 monclient: dump: 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.727 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:50.728 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:50.728 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:50.728 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:50.728 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.745+0000 7f027e049640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.745+0000 7f027e049640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.745+0000 7f027e049640 4 set_mon_vals no callback set 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.746+0000 7f027e049640 10 monclient: shutdown 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.746+0000 7f027e049640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.746+0000 7f027e049640 1 -- 172.21.15.79:0/3125580115 >> v1:172.21.15.79:6789/0 conn(0x7f02780b1f60 legacy=0x7f0278143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.746+0000 7f027e049640 1 -- 172.21.15.79:0/3125580115 shutdown_connections 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.746+0000 7f027e049640 1 -- 172.21.15.79:0/3125580115 >> 172.21.15.79:0/3125580115 conn(0x7f0278066a00 msgr2=0x7f0278066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:50.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.747+0000 7f027e049640 1 -- 172.21.15.79:0/3125580115 shutdown_connections 2024-03-29T10:48:50.733 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.751+0000 7f027e049640 1 -- 172.21.15.79:0/3125580115 wait complete. 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.752+0000 7f027e049640 10 monclient: build_initial_monmap 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.752+0000 7f027e049640 10 monclient: monmap: 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:50.734 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:50.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.752+0000 7f027e049640 1 librados: starting msgr at 2024-03-29T10:48:50.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.752+0000 7f027e049640 1 librados: starting objecter 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.752+0000 7f027e049640 1 Processor -- start 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.753+0000 7f027e049640 1 -- start start 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.753+0000 7f027e049640 1 librados: setting wanted keys 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.753+0000 7f027e049640 1 librados: calling monclient init 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.753+0000 7f027e049640 10 monclient: init 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient(hunting): picked mon.noname-b con 0x7f02780b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:50.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient(hunting): picked mon.noname-c con 0x7f0278143d10 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:50.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient(hunting): picked mon.noname-a con 0x7f0278071610 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:50.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0278142020 con 0x7f0278071610 2024-03-29T10:48:50.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0278070e30 con 0x7f02780b1f60 2024-03-29T10:48:50.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02781dac10 con 0x7f0278143d10 2024-03-29T10:48:50.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:50.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.754+0000 7f027e049640 10 monclient(hunting): authenticate will time out at 1050.046350s 2024-03-29T10:48:50.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.757+0000 7f02777fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f02780b1f60 0x7f0278154cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:50615/0 (socket says 172.21.15.79:50615) 2024-03-29T10:48:50.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.757+0000 7f02777fe640 1 -- 172.21.15.79:0/2673554853 learned_addr learned my addr 172.21.15.79:0/2673554853 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:50.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.757+0000 7f025ffff640 10 client.?.objecter ms_handle_connect 0x7f02780b1f60 2024-03-29T10:48:50.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.757+0000 7f025ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.757+0000 7f025ffff640 10 client.?.objecter ms_handle_connect 0x7f0278143d10 2024-03-29T10:48:50.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.757+0000 7f025ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.759+0000 7f025ffff640 10 client.?.objecter ms_handle_connect 0x7f0278071610 2024-03-29T10:48:50.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.759+0000 7f025ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.759+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160053667 0 0) 0x7f0278142020 con 0x7f0278071610 2024-03-29T10:48:50.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.760+0000 7f025ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.760+0000 7f025ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.760+0000 7f025ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.760+0000 7f025ffff640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:48:50.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.760+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0264004eb0 con 0x7f0278071610 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049931160 0 0) 0x7f02781dac10 con 0x7f0278143d10 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:48:50.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02640051d0 con 0x7f0278143d10 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841709396 0 0) 0x7f0264004eb0 con 0x7f0278071610 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02781dac10 con 0x7f0278071610 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094290752 0 0) 0x7f0278070e30 con 0x7f02780b1f60 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.763+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02640058d0 con 0x7f02780b1f60 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f026c001710 con 0x7f0278071610 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:50.752 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 953200514 0 0) 0x7f02781dac10 con 0x7f0278071610 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 >> v1:172.21.15.79:6790/0 conn(0x7f0278143d10 legacy=0x7f027806f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 >> v1:172.21.15.82:6789/0 conn(0x7f02780b1f60 legacy=0x7f0278154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 1 monclient: found mon.a 2024-03-29T10:48:50.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0278142270 con 0x7f0278071610 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f025ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 10 monclient: _renew_subs 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02781554f0 con 0x7f0278071610 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:50.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 10 monclient: _renew_subs 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f027806ff70 con 0x7f0278071610 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.764+0000 7f027e049640 1 librados: init done 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f026c001d20 con 0x7f0278071610 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f025ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f026c003790 con 0x7f0278071610 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f025ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f025ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f025ffff640 10 monclient: dump: 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.755 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.765+0000 7f0275ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f026c019760 con 0x7f0278071610 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f027e049640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f027e049640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f027e049640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0258005080 con 0x7f0278071610 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f026c083730 con 0x7f0278071610 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f025ffff640 10 client.4266.objecter ms_dispatch 0x7f0278000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f025ffff640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:50.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.767+0000 7f025ffff640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.768+0000 7f025ffff640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.768+0000 7f02777fe640 10 monclient: get_auth_request con 0x7f026406dfa0 auth_method 0 2024-03-29T10:48:50.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.771+0000 7f025ffff640 10 client.4266.objecter ms_handle_connect 0x7f026406dfa0 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.776+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f026c04f010 con 0x7f0278071610 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.776+0000 7f025ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:50.776+0000 7f025ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:51.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.116+0000 7f027e049640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:51.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.116+0000 7f027e049640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:51.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.116+0000 7f027e049640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:51.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.116+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0258005330 con 0x7f0278071610 2024-03-29T10:48:51.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.118+0000 7f025ffff640 1 -- 172.21.15.79:0/2673554853 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f026c04f010 con 0x7f0278071610 2024-03-29T10:48:51.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.118+0000 7f025ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:51.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.118+0000 7f025ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:51.103 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:51.103 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:51.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 10 librados: watch_flush enter 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 10 librados: watch_flush exit 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 10 client.4266.objecter successfully canceled tick 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 >> v1:172.21.15.79:6816/4290694627 conn(0x7f026406dfa0 legacy=0x7f0264070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 10 monclient: shutdown 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 >> v1:172.21.15.79:6789/0 conn(0x7f0278071610 legacy=0x7f02781df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 shutdown_connections 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.124+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 >> 172.21.15.79:0/2673554853 conn(0x7f0278066a00 msgr2=0x7f02780b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.125+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 shutdown_connections 2024-03-29T10:48:51.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.125+0000 7f027e049640 1 -- 172.21.15.79:0/2673554853 wait complete. 2024-03-29T10:48:51.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:51.125+0000 7f027e049640 1 librados: shutdown 2024-03-29T10:48:51.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:57.932 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:58.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.118+0000 7fa48e74d640 10 monclient: get_monmap_and_config 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.118+0000 7fa48e74d640 10 monclient: build_initial_monmap 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.118+0000 7fa48e74d640 10 monclient: monmap: 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:48:58.119556+0000 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:48:58.119556+0000 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:48:58.102 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.120+0000 7fa48e74d640 1 Processor -- start 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.120+0000 7fa48e74d640 1 -- start start 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.120+0000 7fa48e74d640 10 monclient: init 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4880acf60 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4880ad5c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient(hunting): picked mon.noname-a con 0x7fa48813a000 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880aec00 con 0x7fa48813a000 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48813daa0 con 0x7fa4880ad5c0 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880ad9d0 con 0x7fa4880acf60 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:58.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.121+0000 7fa48e74d640 10 monclient(hunting): authenticate will time out at 1057.413767s 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.127+0000 7fa4877fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa4880ad5c0 0x7fa4881398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:45587/0 (socket says 172.21.15.79:45587) 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.127+0000 7fa4877fe640 1 -- 172.21.15.79:0/595594371 learned_addr learned my addr 172.21.15.79:0/595594371 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235175197 0 0) 0x7fa48813daa0 con 0x7fa4880ad5c0 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa470003520 con 0x7fa4880ad5c0 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844341887 0 0) 0x7fa4880ad9d0 con 0x7fa4880acf60 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.128+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48813daa0 con 0x7fa4880acf60 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3372461241 0 0) 0x7fa470003520 con 0x7fa4880ad5c0 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 >> v1:172.21.15.79:6790/0 conn(0x7fa4880acf60 legacy=0x7fa48813ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 >> v1:172.21.15.79:6789/0 conn(0x7fa48813a000 legacy=0x7fa48814b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:58.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 1 monclient: found mon.noname-c 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880b4310 con 0x7fa4880ad5c0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.129+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa478001690 con 0x7fa4880ad5c0 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: dump: 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.115 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4780018b0 con 0x7fa4880ad5c0 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 1 -- 172.21.15.79:0/595594371 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa478001c40 con 0x7fa4880ad5c0 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa486ffd640 10 monclient: dump: 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.116 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa48e74d640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa48e74d640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.130+0000 7fa48e74d640 4 set_mon_vals no callback set 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.131+0000 7fa48e74d640 10 monclient: shutdown 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.131+0000 7fa48e74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:58.117 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.131+0000 7fa48e74d640 1 -- 172.21.15.79:0/595594371 >> v1:172.21.15.82:6789/0 conn(0x7fa4880ad5c0 legacy=0x7fa4881398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.136+0000 7fa48e74d640 1 -- 172.21.15.79:0/595594371 shutdown_connections 2024-03-29T10:48:58.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.136+0000 7fa48e74d640 1 -- 172.21.15.79:0/595594371 >> 172.21.15.79:0/595594371 conn(0x7fa4880b35e0 msgr2=0x7fa4880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:58.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.136+0000 7fa48e74d640 1 -- 172.21.15.79:0/595594371 shutdown_connections 2024-03-29T10:48:58.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.136+0000 7fa48e74d640 1 -- 172.21.15.79:0/595594371 wait complete. 2024-03-29T10:48:58.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.137+0000 7fa48e74d640 10 monclient: build_initial_monmap 2024-03-29T10:48:58.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.137+0000 7fa48e74d640 10 monclient: monmap: 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:58.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 1 librados: starting msgr at 2024-03-29T10:48:58.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 1 librados: starting objecter 2024-03-29T10:48:58.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 1 Processor -- start 2024-03-29T10:48:58.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 1 -- start start 2024-03-29T10:48:58.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 1 librados: setting wanted keys 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 1 librados: calling monclient init 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.138+0000 7fa48e74d640 10 monclient: init 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 10 monclient(hunting): picked mon.noname-c con 0x7fa4880acf60 addr v1:172.21.15.79:6790/0 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 10 monclient(hunting): picked mon.noname-b con 0x7fa4880ad5c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 10 monclient(hunting): picked mon.noname-a con 0x7fa48813a000 addr v1:172.21.15.79:6789/0 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48813c960 con 0x7fa48813a000 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48813d610 con 0x7fa4880ad5c0 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.139+0000 7fa48e74d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48813d8f0 con 0x7fa4880acf60 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.140+0000 7fa48e74d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:58.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.140+0000 7fa48e74d640 10 monclient(hunting): authenticate will time out at 1057.432326s 2024-03-29T10:48:58.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.144+0000 7fa48ccea640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fa48813a000 0x7fa488070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:34337/0 (socket says 172.21.15.79:34337) 2024-03-29T10:48:58.127 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.145+0000 7fa48ccea640 1 -- 172.21.15.79:0/1758738747 learned_addr learned my addr 172.21.15.79:0/1758738747 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.145+0000 7fa484ff9640 10 client.?.objecter ms_handle_connect 0x7fa48813a000 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.145+0000 7fa484ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389925162 0 0) 0x7fa48813c960 con 0x7fa48813a000 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:48:58.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa474004eb0 con 0x7fa48813a000 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362781920 0 0) 0x7fa474004eb0 con 0x7fa48813a000 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa474005300 con 0x7fa48813a000 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa480001a90 con 0x7fa48813a000 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:58.129 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.146+0000 7fa484ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 10 client.?.objecter ms_handle_connect 0x7fa4880acf60 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262663084 0 0) 0x7fa48813d8f0 con 0x7fa4880acf60 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:58.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.147+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4740058a0 con 0x7fa4880acf60 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4264249084 0 0) 0x7fa474005300 con 0x7fa48813a000 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 >> v1:172.21.15.79:6790/0 conn(0x7fa4880acf60 legacy=0x7fa48813bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 >> v1:172.21.15.82:6789/0 conn(0x7fa4880ad5c0 legacy=0x7fa488142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 1 monclient: found mon.a 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:58.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa488143350 con 0x7fa48813a000 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa484ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 10 monclient: _renew_subs 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa488143600 con 0x7fa48813a000 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 10 monclient: _renew_subs 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48813b750 con 0x7fa48813a000 2024-03-29T10:48:58.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.148+0000 7fa48e74d640 1 librados: init done 2024-03-29T10:48:58.134 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa48e74d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:58.134 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa48e74d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:58.134 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa48e74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:58.134 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48813c680 con 0x7fa48813a000 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa480002060 con 0x7fa48813a000 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa484ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa480001a90 con 0x7fa48813a000 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa484ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa484ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa484ff9640 10 monclient: dump: 2024-03-29T10:48:58.135 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:48:58.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.152+0000 7fa4867fc640 4 set_mon_vals no callback set 2024-03-29T10:48:58.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.154+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fa48001ab10 con 0x7fa48813a000 2024-03-29T10:48:58.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.155+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fa480054650 con 0x7fa48813a000 2024-03-29T10:48:58.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.155+0000 7fa484ff9640 10 client.4269.objecter ms_dispatch 0x7fa488000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:58.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.155+0000 7fa484ff9640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:58.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.155+0000 7fa484ff9640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:58.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.155+0000 7fa484ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T10:48:58.139 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.157+0000 7fa487fff640 10 monclient: get_auth_request con 0x7fa47406e410 auth_method 0 2024-03-29T10:48:58.140 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.158+0000 7fa484ff9640 10 client.4269.objecter ms_handle_connect 0x7fa47406e410 2024-03-29T10:48:58.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.163+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4800508f0 con 0x7fa48813a000 2024-03-29T10:48:58.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.163+0000 7fa484ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:58.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.163+0000 7fa484ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:58.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.524+0000 7fa48e74d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.524+0000 7fa48e74d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.524+0000 7fa48e74d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:48:58.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.524+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa488075f10 con 0x7fa48813a000 2024-03-29T10:48:58.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.528+0000 7fa484ff9640 1 -- 172.21.15.79:0/1758738747 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fa480050a90 con 0x7fa48813a000 2024-03-29T10:48:58.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.528+0000 7fa484ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.528+0000 7fa484ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:58.510 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:48:58.510 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:58.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.533+0000 7fa48e74d640 10 librados: watch_flush enter 2024-03-29T10:48:58.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.533+0000 7fa48e74d640 10 librados: watch_flush exit 2024-03-29T10:48:58.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.533+0000 7fa48e74d640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T10:48:58.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.533+0000 7fa48e74d640 10 client.4269.objecter successfully canceled tick 2024-03-29T10:48:58.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.536+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 >> v1:172.21.15.79:6816/4290694627 conn(0x7fa47406e410 legacy=0x7fa4740708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.536+0000 7fa48e74d640 10 monclient: shutdown 2024-03-29T10:48:58.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.536+0000 7fa48e74d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:58.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.536+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 >> v1:172.21.15.79:6789/0 conn(0x7fa48813a000 legacy=0x7fa488070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.536+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 shutdown_connections 2024-03-29T10:48:58.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.536+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 >> 172.21.15.79:0/1758738747 conn(0x7fa4880b35e0 msgr2=0x7fa4880b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:58.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.537+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 shutdown_connections 2024-03-29T10:48:58.519 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.537+0000 7fa48e74d640 1 -- 172.21.15.79:0/1758738747 wait complete. 2024-03-29T10:48:58.519 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:48:58.537+0000 7fa48e74d640 1 librados: shutdown 2024-03-29T10:48:58.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:05.346 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:05.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.527+0000 7f01aa65c640 10 monclient: get_monmap_and_config 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.527+0000 7f01aa65c640 10 monclient: build_initial_monmap 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.527+0000 7f01aa65c640 10 monclient: monmap: 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:05.528404+0000 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:05.528404+0000 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:05.510 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:05.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.528+0000 7f01aa65c640 1 Processor -- start 2024-03-29T10:49:05.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.528+0000 7f01aa65c640 1 -- start start 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.528+0000 7f01aa65c640 10 monclient: init 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient(hunting): picked mon.noname-c con 0x7f01a41422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient(hunting): picked mon.noname-a con 0x7f01a4142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient(hunting): picked mon.noname-b con 0x7f01a40b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a4142c80 con 0x7f01a4142690 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a40aae40 con 0x7f01a41422b0 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a413fbd0 con 0x7f01a40b17d0 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.529+0000 7f01aa65c640 10 monclient(hunting): authenticate will time out at 1064.822045s 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a8bf9640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f01a40b17d0 0x7f01a40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:34148/0 (socket says 172.21.15.79:34148) 2024-03-29T10:49:05.512 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a8bf9640 1 -- 172.21.15.79:0/785579690 learned_addr learned my addr 172.21.15.79:0/785579690 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161886025 0 0) 0x7f01a40aae40 con 0x7f01a41422b0 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a2ffd640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.530+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003520 con 0x7f01a41422b0 2024-03-29T10:49:05.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245790102 0 0) 0x7f01a4142c80 con 0x7f01a4142690 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a40aae40 con 0x7f01a4142690 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1074244897 0 0) 0x7f0190003520 con 0x7f01a41422b0 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 >> v1:172.21.15.79:6790/0 conn(0x7f01a40b17d0 legacy=0x7f01a40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 >> v1:172.21.15.79:6789/0 conn(0x7f01a4142690 legacy=0x7f01a40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a4151940 con 0x7f01a41422b0 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:05.515 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01aa65c640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01aa65c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0194001610 con 0x7f01a41422b0 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: dump: 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:05.516 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01aa65c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0194001770 con 0x7f01a41422b0 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01aa65c640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.531+0000 7f01aa65c640 4 set_mon_vals no callback set 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01a2ffd640 1 -- 172.21.15.79:0/785579690 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0194001bc0 con 0x7f01a41422b0 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01a2ffd640 10 monclient: dump: 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.517 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 10 monclient: shutdown 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 1 -- 172.21.15.79:0/785579690 >> v1:172.21.15.82:6789/0 conn(0x7f01a41422b0 legacy=0x7f01a413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 1 -- 172.21.15.79:0/785579690 shutdown_connections 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 1 -- 172.21.15.79:0/785579690 >> 172.21.15.79:0/785579690 conn(0x7f01a4066a00 msgr2=0x7f01a4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 1 -- 172.21.15.79:0/785579690 shutdown_connections 2024-03-29T10:49:05.518 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.532+0000 7f01aa65c640 1 -- 172.21.15.79:0/785579690 wait complete. 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.534+0000 7f01aa65c640 10 monclient: build_initial_monmap 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.534+0000 7f01aa65c640 10 monclient: monmap: 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.534+0000 7f01aa65c640 1 librados: starting msgr at 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.534+0000 7f01aa65c640 1 librados: starting objecter 2024-03-29T10:49:05.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.534+0000 7f01aa65c640 1 Processor -- start 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.535+0000 7f01aa65c640 1 -- start start 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.535+0000 7f01aa65c640 1 librados: setting wanted keys 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.535+0000 7f01aa65c640 1 librados: calling monclient init 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.535+0000 7f01aa65c640 10 monclient: init 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient(hunting): picked mon.noname-b con 0x7f01a40b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient(hunting): picked mon.noname-a con 0x7f01a41422b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient(hunting): picked mon.noname-c con 0x7f01a4142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a40ad4c0 con 0x7f01a41422b0 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a40ad7a0 con 0x7f01a40b17d0 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a40ada80 con 0x7f01a4142690 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01aa65c640 10 monclient(hunting): authenticate will time out at 1064.828970s 2024-03-29T10:49:05.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01a3fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f01a40b17d0 0x7f01a40ae4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:44494/0 (socket says 172.21.15.79:44494) 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.536+0000 7f01a3fff640 1 -- 172.21.15.79:0/464438665 learned_addr learned my addr 172.21.15.79:0/464438665 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 client.?.objecter ms_handle_connect 0x7f01a40b17d0 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550025046 0 0) 0x7f01a40ad7a0 con 0x7f01a40b17d0 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0184004eb0 con 0x7f01a40b17d0 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 client.?.objecter ms_handle_connect 0x7f01a41422b0 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 client.?.objecter ms_handle_connect 0x7f01a4142690 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.537+0000 7f01a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892838898 0 0) 0x7f0184004eb0 con 0x7f01a40b17d0 2024-03-29T10:49:05.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a40ad7a0 con 0x7f01a40b17d0 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0194001bc0 con 0x7f01a40b17d0 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:05.527 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850904649 0 0) 0x7f01a40ada80 con 0x7f01a4142690 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0184004eb0 con 0x7f01a4142690 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.538+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853362017 0 0) 0x7f01a40ad4c0 con 0x7f01a41422b0 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a40ada80 con 0x7f01a41422b0 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1784638158 0 0) 0x7f01a40ad7a0 con 0x7f01a40b17d0 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 >> v1:172.21.15.79:6790/0 conn(0x7f01a4142690 legacy=0x7f01a41e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 >> v1:172.21.15.79:6789/0 conn(0x7f01a41422b0 legacy=0x7f01a40aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 monclient: found mon.b 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a40adea0 con 0x7f01a40b17d0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 10 monclient: _renew_subs 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a40abb00 con 0x7f01a40b17d0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 10 monclient: _renew_subs 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a40b6be0 con 0x7f01a40b17d0 2024-03-29T10:49:05.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01aa65c640 1 librados: init done 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01940038a0 con 0x7f01a40b17d0 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0194003c70 con 0x7f01a40b17d0 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a27fc640 4 set_mon_vals no callback set 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.539+0000 7f01a0ff9640 10 monclient: dump: 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:05.530 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.541+0000 7f01aa65c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.541+0000 7f01aa65c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.541+0000 7f01aa65c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.541+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0188005080 con 0x7f01a40b17d0 2024-03-29T10:49:05.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f0194003dd0 con 0x7f01a40b17d0 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f0194084c40 con 0x7f01a40b17d0 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a0ff9640 10 client.4252.objecter ms_dispatch 0x7f01a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a0ff9640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a0ff9640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a0ff9640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.542+0000 7f01a37fe640 10 monclient: get_auth_request con 0x7f018406dea0 auth_method 0 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.543+0000 7f01a0ff9640 10 client.4252.objecter ms_handle_connect 0x7f018406dea0 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.548+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0194050520 con 0x7f01a40b17d0 2024-03-29T10:49:05.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.549+0000 7f01a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.533 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:05.549+0000 7f01a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.005+0000 7f01aa65c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.005+0000 7f01aa65c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.005+0000 7f01aa65c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.005+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0188005330 con 0x7f01a40b17d0 2024-03-29T10:49:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.007+0000 7f01a0ff9640 1 -- 172.21.15.79:0/464438665 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f0194052450 con 0x7f01a40b17d0 2024-03-29T10:49:05.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.007+0000 7f01a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.007+0000 7f01a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:05.989 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:05.989 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 10 librados: watch_flush enter 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 10 librados: watch_flush exit 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 10 client.4252.objecter successfully canceled tick 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 >> v1:172.21.15.79:6816/4290694627 conn(0x7f018406dea0 legacy=0x7f0184070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 10 monclient: shutdown 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 >> v1:172.21.15.82:6789/0 conn(0x7f01a40b17d0 legacy=0x7f01a40ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 shutdown_connections 2024-03-29T10:49:05.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.011+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 >> 172.21.15.79:0/464438665 conn(0x7f01a4066a00 msgr2=0x7f01a40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:05.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.012+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 shutdown_connections 2024-03-29T10:49:05.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.012+0000 7f01aa65c640 1 -- 172.21.15.79:0/464438665 wait complete. 2024-03-29T10:49:05.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:06.012+0000 7f01aa65c640 1 librados: shutdown 2024-03-29T10:49:06.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:12.817 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.966+0000 7f229a896640 10 monclient: get_monmap_and_config 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.966+0000 7f229a896640 10 monclient: build_initial_monmap 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.966+0000 7f229a896640 10 monclient: monmap: 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:12.968165+0000 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:12.968165+0000 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:12.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.967+0000 7f229a896640 1 Processor -- start 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.967+0000 7f229a896640 1 -- start start 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.967+0000 7f229a896640 10 monclient: init 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient(hunting): picked mon.noname-b con 0x7f229406ebd0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient(hunting): picked mon.noname-c con 0x7f229406efb0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient(hunting): picked mon.noname-a con 0x7f229413ed90 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294070b40 con 0x7f229413ed90 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294070e20 con 0x7f229406efb0 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294071100 con 0x7f229406ebd0 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:12.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f229a896640 10 monclient(hunting): authenticate will time out at 1072.261585s 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2293fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f229406ebd0 0x7f229413b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35026/0 (socket says 172.21.15.79:35026) 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2293fff640 1 -- 172.21.15.79:0/3911285363 learned_addr learned my addr 172.21.15.79:0/3911285363 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149564559 0 0) 0x7f2294071100 con 0x7f229406ebd0 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f227c004630 con 0x7f229406ebd0 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892710056 0 0) 0x7f227c004630 con 0x7f229406ebd0 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.968+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 >> v1:172.21.15.82:6789/0 conn(0x7f229406efb0 legacy=0x7f229413e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:12.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.969+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 >> v1:172.21.15.79:6789/0 conn(0x7f229413ed90 legacy=0x7f2294149d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:12.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:12.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 1 monclient: found mon.noname-b 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229413c040 con 0x7f229406ebd0 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2288001690 con 0x7f229406ebd0 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: dump: 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.955 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f229a896640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f229a896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22880018f0 con 0x7f229406ebd0 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:12.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 1 -- 172.21.15.79:0/3911285363 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2288001d40 con 0x7f229406ebd0 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f2292ffd640 10 monclient: dump: 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:12.957 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f229a896640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.970+0000 7f229a896640 4 set_mon_vals no callback set 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.971+0000 7f229a896640 10 monclient: shutdown 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.971+0000 7f229a896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.971+0000 7f229a896640 1 -- 172.21.15.79:0/3911285363 >> v1:172.21.15.79:6790/0 conn(0x7f229406ebd0 legacy=0x7f229413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.972+0000 7f229a896640 1 -- 172.21.15.79:0/3911285363 shutdown_connections 2024-03-29T10:49:12.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.972+0000 7f229a896640 1 -- 172.21.15.79:0/3911285363 >> 172.21.15.79:0/3911285363 conn(0x7f22940b4200 msgr2=0x7f22940b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:12.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.972+0000 7f229a896640 1 -- 172.21.15.79:0/3911285363 shutdown_connections 2024-03-29T10:49:12.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.972+0000 7f229a896640 1 -- 172.21.15.79:0/3911285363 wait complete. 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.973+0000 7f229a896640 10 monclient: build_initial_monmap 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.973+0000 7f229a896640 10 monclient: monmap: 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:12.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.973+0000 7f229a896640 1 librados: starting msgr at 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.973+0000 7f229a896640 1 librados: starting objecter 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.973+0000 7f229a896640 1 Processor -- start 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.974+0000 7f229a896640 1 -- start start 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.974+0000 7f229a896640 1 librados: setting wanted keys 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.974+0000 7f229a896640 1 librados: calling monclient init 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.974+0000 7f229a896640 10 monclient: init 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.974+0000 7f229a896640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 10 monclient(hunting): picked mon.noname-a con 0x7f229406ebd0 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 10 monclient(hunting): picked mon.noname-b con 0x7f229406efb0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 10 monclient(hunting): picked mon.noname-c con 0x7f229413ed90 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294140650 con 0x7f229406ebd0 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f2298e33640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f229413ed90 0x7f22941df6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35520/0 (socket says 172.21.15.79:35520) 2024-03-29T10:49:12.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f2298e33640 1 -- 172.21.15.79:0/3376386879 learned_addr learned my addr 172.21.15.79:0/3376386879 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294142490 con 0x7f229406efb0 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f2290ff9640 10 client.?.objecter ms_handle_connect 0x7f229413ed90 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f2290ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2294140ae0 con 0x7f229413ed90 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f2290ff9640 10 client.?.objecter ms_handle_connect 0x7f229406efb0 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f2290ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.975+0000 7f229a896640 10 monclient(hunting): authenticate will time out at 1072.268784s 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.976+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071255946 0 0) 0x7f2294140ae0 con 0x7f229413ed90 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.976+0000 7f2290ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.976+0000 7f2290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.976+0000 7f2290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.976+0000 7f2290ff9640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:49:12.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.976+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2280004eb0 con 0x7f229413ed90 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304225348 0 0) 0x7f2280004eb0 con 0x7f229413ed90 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2294140ae0 con 0x7f229413ed90 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f228c001ad0 con 0x7f229413ed90 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:12.969 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638183696 0 0) 0x7f2294142490 con 0x7f229406efb0 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2280004eb0 con 0x7f229406efb0 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 728575324 0 0) 0x7f2280004eb0 con 0x7f229406efb0 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2294142490 con 0x7f229406efb0 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1714682556 0 0) 0x7f2294140ae0 con 0x7f229413ed90 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.977+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 >> v1:172.21.15.82:6789/0 conn(0x7f229406efb0 legacy=0x7f22941d4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 >> v1:172.21.15.79:6789/0 conn(0x7f229406ebd0 legacy=0x7f2294140f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:12.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 1 monclient: found mon.c 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22941cc5c0 con 0x7f229413ed90 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f2290ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f229a896640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f229a896640 10 monclient: _renew_subs 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f229a896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.978+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22941400a0 con 0x7f229413ed90 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f229a896640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f229a896640 10 monclient: _renew_subs 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f229a896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:12.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22941d5580 con 0x7f229413ed90 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f229a896640 1 librados: init done 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f228c001c30 con 0x7f229413ed90 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f2290ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f228c004d80 con 0x7f229413ed90 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f2290ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.979+0000 7f2290ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.980+0000 7f2290ff9640 10 monclient: dump: 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:12.972 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.980+0000 7f22927fc640 4 set_mon_vals no callback set 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.980+0000 7f229a896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f229a896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f229a896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2274005080 con 0x7f229413ed90 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f228c01ad50 con 0x7f229413ed90 2024-03-29T10:49:12.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f228c084c40 con 0x7f229413ed90 2024-03-29T10:49:12.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f2290ff9640 10 client.4280.objecter ms_dispatch 0x7f2294000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:12.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f2290ff9640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:12.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.981+0000 7f2290ff9640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:12.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.982+0000 7f2290ff9640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T10:49:12.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.982+0000 7f2293fff640 10 monclient: get_auth_request con 0x7f228006dfa0 auth_method 0 2024-03-29T10:49:12.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.982+0000 7f2290ff9640 10 client.4280.objecter ms_handle_connect 0x7f228006dfa0 2024-03-29T10:49:12.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.988+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f228c050520 con 0x7f229413ed90 2024-03-29T10:49:12.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.989+0000 7f2290ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:12.989+0000 7f2290ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:13.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.297+0000 7f229a896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:13.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.298+0000 7f229a896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:13.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.298+0000 7f229a896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:13.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.298+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2274005330 con 0x7f229413ed90 2024-03-29T10:49:13.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.299+0000 7f2290ff9640 1 -- 172.21.15.79:0/3376386879 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f228c052450 con 0x7f229413ed90 2024-03-29T10:49:13.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.299+0000 7f2290ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:13.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.299+0000 7f2290ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:13.282 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:13.282 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:13.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 10 librados: watch_flush enter 2024-03-29T10:49:13.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 10 librados: watch_flush exit 2024-03-29T10:49:13.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 10 client.4280.objecter successfully canceled tick 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 >> v1:172.21.15.79:6816/4290694627 conn(0x7f228006dfa0 legacy=0x7f2280070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 10 monclient: shutdown 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 >> v1:172.21.15.79:6790/0 conn(0x7f229413ed90 legacy=0x7f22941df6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 shutdown_connections 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 >> 172.21.15.79:0/3376386879 conn(0x7f22940b4200 msgr2=0x7f229413b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 shutdown_connections 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 -- 172.21.15.79:0/3376386879 wait complete. 2024-03-29T10:49:13.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:13.302+0000 7f229a896640 1 librados: shutdown 2024-03-29T10:49:13.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:20.107 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:20.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.258+0000 7fcadbaba640 10 monclient: get_monmap_and_config 2024-03-29T10:49:20.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.258+0000 7fcadbaba640 10 monclient: build_initial_monmap 2024-03-29T10:49:20.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.258+0000 7fcadbaba640 10 monclient: monmap: 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:20.259596+0000 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:20.259596+0000 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:20.241 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:20.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.259+0000 7fcadbaba640 1 Processor -- start 2024-03-29T10:49:20.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.259+0000 7fcadbaba640 1 -- start start 2024-03-29T10:49:20.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.259+0000 7fcadbaba640 10 monclient: init 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient(hunting): picked mon.noname-c con 0x7fcad40b0770 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient(hunting): picked mon.noname-a con 0x7fcad414a700 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient(hunting): picked mon.noname-b con 0x7fcad40ac5a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad40aba70 con 0x7fcad414a700 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad40aecb0 con 0x7fcad40b0770 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad40af150 con 0x7fcad40ac5a0 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.260+0000 7fcadbaba640 10 monclient(hunting): authenticate will time out at 1079.553047s 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.261+0000 7fcad9856640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fcad40b0770 0x7fcad4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:55304/0 (socket says 172.21.15.79:55304) 2024-03-29T10:49:20.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.261+0000 7fcad9856640 1 -- 172.21.15.79:0/2065247296 learned_addr learned my addr 172.21.15.79:0/2065247296 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910087813 0 0) 0x7fcad40aecb0 con 0x7fcad40b0770 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac0003520 con 0x7fcad40b0770 2024-03-29T10:49:20.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590537402 0 0) 0x7fcad40af150 con 0x7fcad40ac5a0 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad40aecb0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4120697250 0 0) 0x7fcad40aecb0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 >> v1:172.21.15.82:6789/0 conn(0x7fcad40b0770 legacy=0x7fcad4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.262+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 >> v1:172.21.15.79:6789/0 conn(0x7fcad414a700 legacy=0x7fcad40abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: _finish_hunting 0 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 1 monclient: found mon.noname-b 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:49:20.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad40ac9d0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: _finish_auth 0 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: _check_auth_tickets 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcac40015f0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: dump: 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:20.248 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcadbaba640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcadbaba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcac4001750 con 0x7fcad40ac5a0 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 1 -- 172.21.15.79:0/2065247296 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcac4001ba0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcad8854640 10 monclient: dump: 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.249 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcadbaba640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.263+0000 7fcadbaba640 4 set_mon_vals no callback set 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.264+0000 7fcadbaba640 10 monclient: shutdown 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.264+0000 7fcadbaba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.264+0000 7fcadbaba640 1 -- 172.21.15.79:0/2065247296 >> v1:172.21.15.79:6790/0 conn(0x7fcad40ac5a0 legacy=0x7fcad4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.264+0000 7fcadbaba640 1 -- 172.21.15.79:0/2065247296 shutdown_connections 2024-03-29T10:49:20.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.264+0000 7fcadbaba640 1 -- 172.21.15.79:0/2065247296 >> 172.21.15.79:0/2065247296 conn(0x7fcad406b660 msgr2=0x7fcad406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.265+0000 7fcadbaba640 1 -- 172.21.15.79:0/2065247296 shutdown_connections 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.265+0000 7fcadbaba640 1 -- 172.21.15.79:0/2065247296 wait complete. 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 10 monclient: build_initial_monmap 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 10 monclient: monmap: 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:20.258 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 1 librados: starting msgr at 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 1 librados: starting objecter 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 1 Processor -- start 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 1 -- start start 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 1 librados: setting wanted keys 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 1 librados: calling monclient init 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.266+0000 7fcadbaba640 10 monclient: init 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.267+0000 7fcadbaba640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:20.259 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.267+0000 7fcadbaba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.267+0000 7fcadbaba640 10 monclient(hunting): picked mon.noname-c con 0x7fcad40ac5a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcad9856640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fcad40ac5a0 0x7fcad4128090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:57194/0 (socket says 172.21.15.79:57194) 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcad9856640 1 -- 172.21.15.79:0/2089220764 learned_addr learned my addr 172.21.15.79:0/2089220764 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 10 monclient(hunting): picked mon.noname-b con 0x7fcad40b0770 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 10 monclient(hunting): picked mon.noname-a con 0x7fcad414a700 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcaca7fc640 10 client.?.objecter ms_handle_connect 0x7fcad40ac5a0 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad41287a0 con 0x7fcad414a700 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad412a480 con 0x7fcad40b0770 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad4129240 con 0x7fcad40ac5a0 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcadbaba640 10 monclient(hunting): authenticate will time out at 1079.560909s 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcaca7fc640 10 client.?.objecter ms_handle_connect 0x7fcad40b0770 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.268+0000 7fcaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:20.260 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970073695 0 0) 0x7fcad4129240 con 0x7fcad40ac5a0 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab8004eb0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630954911 0 0) 0x7fcad412a480 con 0x7fcad40b0770 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.269+0000 7fcaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad4129240 con 0x7fcad40b0770 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55065068 0 0) 0x7fcab8004eb0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcad412a480 con 0x7fcad40ac5a0 2024-03-29T10:49:20.261 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcacc001c70 con 0x7fcad40ac5a0 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient(hunting): dump: 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:20.262 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3040550848 0 0) 0x7fcad412a480 con 0x7fcad40ac5a0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 >> v1:172.21.15.82:6789/0 conn(0x7fcad40b0770 legacy=0x7fcad41db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 >> v1:172.21.15.79:6789/0 conn(0x7fcad414a700 legacy=0x7fcad41dbe00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.270+0000 7fcaca7fc640 1 monclient: found mon.c 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad4147c30 con 0x7fcad40ac5a0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 10 monclient: _renew_subs 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:20.263 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad4128e60 con 0x7fcad40ac5a0 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 10 monclient: _renew_subs 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad41e71e0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcadbaba640 1 librados: init done 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcacc001dd0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcacc004dc0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcaca7fc640 10 monclient: dump: 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:20.264 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.271+0000 7fcacbfff640 4 set_mon_vals no callback set 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.273+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fcacc004f20 con 0x7fcad40ac5a0 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.273+0000 7fcadbaba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.273+0000 7fcadbaba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.273+0000 7fcadbaba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.273+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcabc005080 con 0x7fcad40ac5a0 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fcacc084e80 con 0x7fcad40ac5a0 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcaca7fc640 10 client.4286.objecter ms_dispatch 0x7fcad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:20.265 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcaca7fc640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcaca7fc640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcad9055640 10 monclient: get_auth_request con 0x7fcab806db30 auth_method 0 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcaca7fc640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.274+0000 7fcaca7fc640 10 client.4286.objecter ms_handle_connect 0x7fcab806db30 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.281+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcacc0505b0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.281+0000 7fcaca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:20.266 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.281+0000 7fcaca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:20.576 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.594+0000 7fcadbaba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.576 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.594+0000 7fcadbaba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.576 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.594+0000 7fcadbaba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:20.576 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.594+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcabc005330 con 0x7fcad40ac5a0 2024-03-29T10:49:20.577 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.595+0000 7fcaca7fc640 1 -- 172.21.15.79:0/2089220764 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fcacc0526b0 con 0x7fcad40ac5a0 2024-03-29T10:49:20.577 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.595+0000 7fcaca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.577 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.595+0000 7fcaca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:20.577 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:20.577 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:20.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 10 librados: watch_flush enter 2024-03-29T10:49:20.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 10 librados: watch_flush exit 2024-03-29T10:49:20.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 10 client.4286.objecter successfully canceled tick 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 >> v1:172.21.15.79:6816/4290694627 conn(0x7fcab806db30 legacy=0x7fcab806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 10 monclient: shutdown 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 >> v1:172.21.15.79:6790/0 conn(0x7fcad40ac5a0 legacy=0x7fcad4128090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 shutdown_connections 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 >> 172.21.15.79:0/2089220764 conn(0x7fcad406b660 msgr2=0x7fcad40b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 shutdown_connections 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 -- 172.21.15.79:0/2089220764 wait complete. 2024-03-29T10:49:20.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:20.599+0000 7fcadbaba640 1 librados: shutdown 2024-03-29T10:49:20.595 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:27.402 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.780+0000 7f608bfff640 10 monclient: get_monmap_and_config 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.780+0000 7f608bfff640 10 monclient: build_initial_monmap 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.780+0000 7f608bfff640 10 monclient: monmap: 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:27.781127+0000 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:27.781127+0000 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:27.762 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:27.763 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:27.763 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:27.763 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:27.763 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.781+0000 7f608bfff640 1 Processor -- start 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.781+0000 7f608bfff640 1 -- start start 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.781+0000 7f608bfff640 10 monclient: init 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f6084006d80 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f60840cc310 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f60840c9430 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084008470 con 0x7f6084006d80 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608402be30 con 0x7f60840cc310 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608402c110 con 0x7f60840c9430 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:27.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.782+0000 7f608bfff640 10 monclient(hunting): authenticate will time out at 1087.074618s 2024-03-29T10:49:27.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.785+0000 7f608affd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f6084006d80 0x7f60840cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:35125/0 (socket says 172.21.15.79:35125) 2024-03-29T10:49:27.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.785+0000 7f608affd640 1 -- 172.21.15.79:0/3451576396 learned_addr learned my addr 172.21.15.79:0/3451576396 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:27.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660371169 0 0) 0x7f6084008470 con 0x7f6084006d80 2024-03-29T10:49:27.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6074003520 con 0x7f6084006d80 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346679896 0 0) 0x7f608402c110 con 0x7f60840c9430 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6084008470 con 0x7f60840c9430 2024-03-29T10:49:27.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463569075 0 0) 0x7f6084008470 con 0x7f60840c9430 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 >> v1:172.21.15.82:6789/0 conn(0x7f60840cc310 legacy=0x7f60840c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 >> v1:172.21.15.79:6789/0 conn(0x7f6084006d80 legacy=0x7f60840cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 monclient: found mon.noname-b 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608402c3f0 con 0x7f60840c9430 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f608c04f730 con 0x7f60840c9430 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:27.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.787+0000 7f6089ffb640 10 monclient: dump: 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:27.771 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f608bfff640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f608bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f608c0008d0 con 0x7f60840c9430 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f6089ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f6089ffb640 1 -- 172.21.15.79:0/3451576396 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f608c0758b0 con 0x7f60840c9430 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f6089ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f6089ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f6089ffb640 10 monclient: dump: 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:27.773 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f608bfff640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.789+0000 7f608bfff640 4 set_mon_vals no callback set 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.790+0000 7f608bfff640 10 monclient: shutdown 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.790+0000 7f608bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.790+0000 7f608bfff640 1 -- 172.21.15.79:0/3451576396 >> v1:172.21.15.79:6790/0 conn(0x7f60840c9430 legacy=0x7f60840d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.791+0000 7f608bfff640 1 -- 172.21.15.79:0/3451576396 shutdown_connections 2024-03-29T10:49:27.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.791+0000 7f608bfff640 1 -- 172.21.15.79:0/3451576396 >> 172.21.15.79:0/3451576396 conn(0x7f6084005c00 msgr2=0x7f6084006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:27.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.791+0000 7f608bfff640 1 -- 172.21.15.79:0/3451576396 shutdown_connections 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.791+0000 7f608bfff640 1 -- 172.21.15.79:0/3451576396 wait complete. 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.792+0000 7f608bfff640 10 monclient: build_initial_monmap 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.792+0000 7f608bfff640 10 monclient: monmap: 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:27.776 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.792+0000 7f608bfff640 1 librados: starting msgr at 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.792+0000 7f608bfff640 1 librados: starting objecter 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.793+0000 7f608bfff640 1 Processor -- start 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.793+0000 7f608bfff640 1 -- start start 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.793+0000 7f608bfff640 1 librados: setting wanted keys 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.793+0000 7f608bfff640 1 librados: calling monclient init 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.793+0000 7f608bfff640 10 monclient: init 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.795+0000 7f608bfff640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:27.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.795+0000 7f608bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:27.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.795+0000 7f608bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f6084006d80 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:27.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.799+0000 7f608bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f60840c9430 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:27.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.799+0000 7f608bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f60840cc310 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:27.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.802+0000 7f608bfff640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608402e010 con 0x7f60840c9430 2024-03-29T10:49:27.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.802+0000 7f608bfff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6084164440 con 0x7f60840cc310 2024-03-29T10:49:27.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.802+0000 7f608bfff640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60840c9a00 con 0x7f6084006d80 2024-03-29T10:49:27.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.802+0000 7f608bfff640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:27.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.802+0000 7f608bfff640 10 monclient(hunting): authenticate will time out at 1087.094527s 2024-03-29T10:49:27.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f608affd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f6084006d80 0x7f6084155120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:40851/0 (socket says 172.21.15.79:40851) 2024-03-29T10:49:27.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f608affd640 1 -- 172.21.15.79:0/3726977250 learned_addr learned my addr 172.21.15.79:0/3726977250 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 10 client.?.objecter ms_handle_connect 0x7f6084006d80 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 10 client.?.objecter ms_handle_connect 0x7f60840cc310 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56689835 0 0) 0x7f60840c9a00 con 0x7f6084006d80 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.805+0000 7f6073fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.806+0000 7f6073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.806+0000 7f6073fff640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.806+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078004eb0 con 0x7f6084006d80 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.806+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513049466 0 0) 0x7f6084164440 con 0x7f60840cc310 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.806+0000 7f6073fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60840c9a00 con 0x7f60840cc310 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 client.?.objecter ms_handle_connect 0x7f60840c9430 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048868757 0 0) 0x7f608402e010 con 0x7f60840c9430 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.807+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6084164440 con 0x7f60840c9430 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786224263 0 0) 0x7f60840c9a00 con 0x7f60840cc310 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f608402e010 con 0x7f60840cc310 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f608c04f400 con 0x7f60840cc310 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 10 monclient(hunting): dump: 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:27.797 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.808+0000 7f6073fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.811+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1849501236 0 0) 0x7f6078004eb0 con 0x7f6084006d80 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60840c9a00 con 0x7f6084006d80 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592883458 0 0) 0x7f6084164440 con 0x7f60840c9430 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6078004eb0 con 0x7f60840c9430 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 993814874 0 0) 0x7f608402e010 con 0x7f60840cc310 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 >> v1:172.21.15.79:6790/0 conn(0x7f6084006d80 legacy=0x7f6084155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 >> v1:172.21.15.79:6789/0 conn(0x7f60840c9430 legacy=0x7f6084164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 monclient: found mon.b 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:27.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840c9b90 con 0x7f60840cc310 2024-03-29T10:49:27.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:27.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:27.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.812+0000 7f6073fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:27.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.813+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f608c04f730 con 0x7f60840cc310 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.813+0000 7f608bfff640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.814+0000 7f608bfff640 10 monclient: _renew_subs 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.814+0000 7f608bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.814+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6084165070 con 0x7f60840cc310 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.814+0000 7f608bfff640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.814+0000 7f608bfff640 10 monclient: _renew_subs 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.814+0000 7f608bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.815+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f608402cd20 con 0x7f60840cc310 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.815+0000 7f608bfff640 1 librados: init done 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.815+0000 7f6073fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:27.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.817+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f608c067d90 con 0x7f60840cc310 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.817+0000 7f6073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.817+0000 7f6073fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.817+0000 7f6073fff640 10 monclient: dump: 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:27.801 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:27.802 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f608c067fb0 con 0x7f60840cc310 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f60897fa640 4 set_mon_vals no callback set 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f608c0fe200 con 0x7f60840cc310 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f6073fff640 10 client.4267.objecter ms_dispatch 0x7f608c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f6073fff640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f6073fff640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.818+0000 7f6073fff640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.819+0000 7f608bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.819+0000 7f608bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.819+0000 7f608bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:27.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.819+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c005080 con 0x7f60840cc310 2024-03-29T10:49:27.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.819+0000 7f608affd640 10 monclient: get_auth_request con 0x7f607806e020 auth_method 0 2024-03-29T10:49:27.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.819+0000 7f6073fff640 10 client.4267.objecter ms_handle_connect 0x7f607806e020 2024-03-29T10:49:27.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.827+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f608c0c9b60 con 0x7f60840cc310 2024-03-29T10:49:27.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.827+0000 7f6073fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:27.827+0000 7f6073fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:28.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.159+0000 7f608bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:28.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.159+0000 7f608bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:28.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.159+0000 7f608bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:28.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.159+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f606c005330 con 0x7f60840cc310 2024-03-29T10:49:28.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.162+0000 7f6073fff640 1 -- 172.21.15.79:0/3726977250 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f608c0cba90 con 0x7f60840cc310 2024-03-29T10:49:28.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.162+0000 7f6073fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:28.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.162+0000 7f6073fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:28.144 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:28.144 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:28.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 10 librados: watch_flush enter 2024-03-29T10:49:28.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 10 librados: watch_flush exit 2024-03-29T10:49:28.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T10:49:28.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 10 client.4267.objecter successfully canceled tick 2024-03-29T10:49:28.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 >> v1:172.21.15.79:6816/4290694627 conn(0x7f607806e020 legacy=0x7f60780704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:28.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 10 monclient: shutdown 2024-03-29T10:49:28.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:28.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.166+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 >> v1:172.21.15.82:6789/0 conn(0x7f60840cc310 legacy=0x7f608416b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:28.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.171+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 shutdown_connections 2024-03-29T10:49:28.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.171+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 >> 172.21.15.79:0/3726977250 conn(0x7f6084005c00 msgr2=0x7f6084039c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:28.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.171+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 shutdown_connections 2024-03-29T10:49:28.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.171+0000 7f608bfff640 1 -- 172.21.15.79:0/3726977250 wait complete. 2024-03-29T10:49:28.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:28.171+0000 7f608bfff640 1 librados: shutdown 2024-03-29T10:49:28.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:34.980 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.146+0000 7fa577082640 10 monclient: get_monmap_and_config 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.146+0000 7fa577082640 10 monclient: build_initial_monmap 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.146+0000 7fa577082640 10 monclient: monmap: 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:35.147202+0000 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:35.147202+0000 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:35.128 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:35.129 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:35.129 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.147+0000 7fa577082640 1 Processor -- start 2024-03-29T10:49:35.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.147+0000 7fa577082640 1 -- start start 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.147+0000 7fa577082640 10 monclient: init 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient(hunting): picked mon.noname-c con 0x7fa5700b2800 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700b3220 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient(hunting): picked mon.noname-a con 0x7fa5701439c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:35.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700b3840 con 0x7fa5701439c0 2024-03-29T10:49:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57006f230 con 0x7fa5700b2800 2024-03-29T10:49:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57006ef50 con 0x7fa5700b3220 2024-03-29T10:49:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:35.132 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.148+0000 7fa577082640 10 monclient(hunting): authenticate will time out at 1094.440479s 2024-03-29T10:49:35.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.151+0000 7fa576080640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa5700b2800 0x7fa5701432b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:41077/0 (socket says 172.21.15.79:41077) 2024-03-29T10:49:35.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.151+0000 7fa576080640 1 -- 172.21.15.79:0/2102415047 learned_addr learned my addr 172.21.15.79:0/2102415047 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:35.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.154+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675155224 0 0) 0x7fa57006f230 con 0x7fa5700b2800 2024-03-29T10:49:35.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.154+0000 7fa57507e640 10 monclient(hunting): _negotiate 2024-03-29T10:49:35.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.154+0000 7fa57507e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:35.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.154+0000 7fa57507e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:35.139 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.154+0000 7fa57507e640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:49:35.139 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.154+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c003520 con 0x7fa5700b2800 2024-03-29T10:49:35.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318140710 0 0) 0x7fa5700b3840 con 0x7fa5701439c0 2024-03-29T10:49:35.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient(hunting): _negotiate 2024-03-29T10:49:35.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:35.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:35.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa57006f230 con 0x7fa5701439c0 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 503890548 0 0) 0x7fa55c003520 con 0x7fa5700b2800 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 >> v1:172.21.15.79:6790/0 conn(0x7fa5700b3220 legacy=0x7fa570146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 >> v1:172.21.15.79:6789/0 conn(0x7fa5701439c0 legacy=0x7fa570151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: _finish_hunting 0 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 monclient: found mon.noname-c 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:35.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570071d30 con 0x7fa5700b2800 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: _finish_auth 0 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: _check_auth_tickets 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa568001a10 con 0x7fa5700b2800 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: dump: 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:35.151 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:35.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa577082640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa577082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa568001b70 con 0x7fa5700b2800 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 1 -- 172.21.15.79:0/2102415047 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa568001f80 con 0x7fa5700b2800 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa577082640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa577082640 4 set_mon_vals no callback set 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:35.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.155+0000 7fa57507e640 10 monclient: dump: 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:35.154 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.156+0000 7fa577082640 10 monclient: shutdown 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.156+0000 7fa577082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.156+0000 7fa577082640 1 -- 172.21.15.79:0/2102415047 >> v1:172.21.15.82:6789/0 conn(0x7fa5700b2800 legacy=0x7fa5701432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.158+0000 7fa577082640 1 -- 172.21.15.79:0/2102415047 shutdown_connections 2024-03-29T10:49:35.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.158+0000 7fa577082640 1 -- 172.21.15.79:0/2102415047 >> 172.21.15.79:0/2102415047 conn(0x7fa570066a00 msgr2=0x7fa570066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.160+0000 7fa577082640 1 -- 172.21.15.79:0/2102415047 shutdown_connections 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.160+0000 7fa577082640 1 -- 172.21.15.79:0/2102415047 wait complete. 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.161+0000 7fa577082640 10 monclient: build_initial_monmap 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.161+0000 7fa577082640 10 monclient: monmap: 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:35.156 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.161+0000 7fa577082640 1 librados: starting msgr at 2024-03-29T10:49:35.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.161+0000 7fa577082640 1 librados: starting objecter 2024-03-29T10:49:35.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.161+0000 7fa577082640 1 Processor -- start 2024-03-29T10:49:35.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.162+0000 7fa577082640 1 -- start start 2024-03-29T10:49:35.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.162+0000 7fa577082640 1 librados: setting wanted keys 2024-03-29T10:49:35.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.162+0000 7fa577082640 1 librados: calling monclient init 2024-03-29T10:49:35.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.162+0000 7fa577082640 10 monclient: init 2024-03-29T10:49:35.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.163+0000 7fa577082640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:35.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.163+0000 7fa577082640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:35.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.163+0000 7fa577082640 10 monclient(hunting): picked mon.noname-c con 0x7fa5700b2800 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:35.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.166+0000 7fa57587f640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa5700b3220 0x7fa57013ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:41167/0 (socket says 172.21.15.79:41167) 2024-03-29T10:49:35.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.166+0000 7fa577082640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700b3220 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:35.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.166+0000 7fa577082640 10 monclient(hunting): picked mon.noname-a con 0x7fa5701439c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:35.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.168+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5701482b0 con 0x7fa5701439c0 2024-03-29T10:49:35.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.168+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57013bb40 con 0x7fa5700b3220 2024-03-29T10:49:35.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.168+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5701dc120 con 0x7fa5700b2800 2024-03-29T10:49:35.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.168+0000 7fa577082640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:35.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.168+0000 7fa577082640 10 monclient(hunting): authenticate will time out at 1094.460238s 2024-03-29T10:49:35.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.167+0000 7fa566ffd640 10 client.?.objecter ms_handle_connect 0x7fa5700b2800 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.168+0000 7fa566ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.167+0000 7fa57587f640 1 -- 172.21.15.79:0/853939182 learned_addr learned my addr 172.21.15.79:0/853939182 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.169+0000 7fa566ffd640 10 client.?.objecter ms_handle_connect 0x7fa5700b3220 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.169+0000 7fa566ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356144897 0 0) 0x7fa57013bb40 con 0x7fa5700b3220 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:35.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa554004eb0 con 0x7fa5700b3220 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456950347 0 0) 0x7fa5701dc120 con 0x7fa5700b2800 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa57013bb40 con 0x7fa5700b2800 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637700118 0 0) 0x7fa554004eb0 con 0x7fa5700b3220 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5701dc120 con 0x7fa5700b3220 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa5600018f0 con 0x7fa5700b3220 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:35.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:35.173 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.170+0000 7fa566ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1686019972 0 0) 0x7fa5701dc120 con 0x7fa5700b3220 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 >> v1:172.21.15.79:6790/0 conn(0x7fa5700b2800 legacy=0x7fa5701dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 >> v1:172.21.15.79:6789/0 conn(0x7fa5701439c0 legacy=0x7fa57013b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 monclient: found mon.b 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570147690 con 0x7fa5700b3220 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:35.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa560003610 con 0x7fa5700b3220 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa560003a20 con 0x7fa5700b3220 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa566ffd640 10 monclient: dump: 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:35.175 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:35.176 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:35.176 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:35.176 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:35.176 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:35.177 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:35.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.171+0000 7fa567fff640 4 set_mon_vals no callback set 2024-03-29T10:49:35.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:49:35.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 10 monclient: _renew_subs 2024-03-29T10:49:35.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570147940 con 0x7fa5700b3220 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 10 monclient: _renew_subs 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.173+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57013bf40 con 0x7fa5700b3220 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.174+0000 7fa577082640 1 librados: init done 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.176+0000 7fa577082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.176+0000 7fa577082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.176+0000 7fa577082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.176+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57004f730 con 0x7fa5700b3220 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.180+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fa560003b80 con 0x7fa5700b3220 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fa5600849a0 con 0x7fa5700b3220 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa566ffd640 10 client.4273.objecter ms_dispatch 0x7fa570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:35.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa566ffd640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa566ffd640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa566ffd640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa576080640 10 monclient: get_auth_request con 0x7fa55406dd40 auth_method 0 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.181+0000 7fa566ffd640 10 client.4273.objecter ms_handle_connect 0x7fa55406dd40 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.188+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa560050280 con 0x7fa5700b3220 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.188+0000 7fa566ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:35.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.188+0000 7fa566ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:35.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.510+0000 7fa577082640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:35.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.510+0000 7fa577082640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:35.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.510+0000 7fa577082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:49:35.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.510+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa57013ea70 con 0x7fa5700b3220 2024-03-29T10:49:35.493 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.511+0000 7fa566ffd640 1 -- 172.21.15.79:0/853939182 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fa5600521b0 con 0x7fa5700b3220 2024-03-29T10:49:35.493 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.511+0000 7fa566ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:35.493 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.511+0000 7fa566ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 10 librados: watch_flush enter 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 10 librados: watch_flush exit 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 10 client.4273.objecter successfully canceled tick 2024-03-29T10:49:35.505 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 >> v1:172.21.15.79:6816/4290694627 conn(0x7fa55406dd40 legacy=0x7fa554070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 10 monclient: shutdown 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 >> v1:172.21.15.82:6789/0 conn(0x7fa5700b3220 legacy=0x7fa57013ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 shutdown_connections 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.515+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 >> 172.21.15.79:0/853939182 conn(0x7fa570066a00 msgr2=0x7fa570069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.516+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 shutdown_connections 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.516+0000 7fa577082640 1 -- 172.21.15.79:0/853939182 wait complete. 2024-03-29T10:49:35.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:35.516+0000 7fa577082640 1 librados: shutdown 2024-03-29T10:49:35.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:42.322 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:42.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.488+0000 7faf3788f640 10 monclient: get_monmap_and_config 2024-03-29T10:49:42.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.488+0000 7faf3788f640 10 monclient: build_initial_monmap 2024-03-29T10:49:42.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.488+0000 7faf3788f640 10 monclient: monmap: 2024-03-29T10:49:42.473 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:42.473 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:42.473 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:42.489861+0000 2024-03-29T10:49:42.473 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:42.489861+0000 2024-03-29T10:49:42.473 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:42.474 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:42.474 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:42.474 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:42.474 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:42.474 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:42.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.489+0000 7faf3788f640 1 Processor -- start 2024-03-29T10:49:42.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.490+0000 7faf3788f640 1 -- start start 2024-03-29T10:49:42.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.490+0000 7faf3788f640 10 monclient: init 2024-03-29T10:49:42.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:42.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:42.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient(hunting): picked mon.noname-a con 0x7faf300b0770 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:42.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient(hunting): picked mon.noname-c con 0x7faf3014a700 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:42.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient(hunting): picked mon.noname-b con 0x7faf300ac5a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:42.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300aba70 con 0x7faf300b0770 2024-03-29T10:49:42.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300aecb0 con 0x7faf3014a700 2024-03-29T10:49:42.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300af150 con 0x7faf300ac5a0 2024-03-29T10:49:42.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:42.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.491+0000 7faf3788f640 10 monclient(hunting): authenticate will time out at 1101.783558s 2024-03-29T10:49:42.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.496+0000 7faf35e2c640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7faf300ac5a0 0x7faf30151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60615/0 (socket says 172.21.15.79:60615) 2024-03-29T10:49:42.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.496+0000 7faf3562b640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7faf300b0770 0x7faf30149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:34461/0 (socket says 172.21.15.79:34461) 2024-03-29T10:49:42.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.496+0000 7faf3562b640 1 -- 172.21.15.79:0/3091657876 learned_addr learned my addr 172.21.15.79:0/3091657876 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:42.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233818792 0 0) 0x7faf300aba70 con 0x7faf300b0770 2024-03-29T10:49:42.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.506 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:49:42.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c003520 con 0x7faf300b0770 2024-03-29T10:49:42.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671381515 0 0) 0x7faf300af150 con 0x7faf300ac5a0 2024-03-29T10:49:42.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.507 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.508 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.508 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:49:42.508 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf300aba70 con 0x7faf300ac5a0 2024-03-29T10:49:42.508 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531065448 0 0) 0x7faf300aecb0 con 0x7faf3014a700 2024-03-29T10:49:42.508 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.508 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf300af150 con 0x7faf3014a700 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218949931 0 0) 0x7faf300aba70 con 0x7faf300ac5a0 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.498+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 >> v1:172.21.15.82:6789/0 conn(0x7faf3014a700 legacy=0x7faf300abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 >> v1:172.21.15.79:6789/0 conn(0x7faf300b0770 legacy=0x7faf30149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.509 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:42.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 1 monclient: found mon.noname-b 2024-03-29T10:49:42.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:42.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:49:42.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300ac9d0 con 0x7faf300ac5a0 2024-03-29T10:49:42.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:42.510 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf3788f640 5 monclient: authenticate success, global_id 4301 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf3788f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faf28001920 con 0x7faf300ac5a0 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:42.511 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: dump: 2024-03-29T10:49:42.512 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:42.512 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:42.512 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.512 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.512 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.512 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf3788f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf28001b40 con 0x7faf300ac5a0 2024-03-29T10:49:42.513 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.499+0000 7faf27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:42.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.500+0000 7faf3788f640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:42.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.500+0000 7faf3788f640 4 set_mon_vals no callback set 2024-03-29T10:49:42.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.500+0000 7faf27fff640 1 -- 172.21.15.79:0/3091657876 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faf28001ed0 con 0x7faf300ac5a0 2024-03-29T10:49:42.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.500+0000 7faf27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.500+0000 7faf27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:42.514 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.500+0000 7faf27fff640 10 monclient: dump: 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:42.515 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:42.516 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:42.516 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:42.516 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:42.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.501+0000 7faf3788f640 10 monclient: shutdown 2024-03-29T10:49:42.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.501+0000 7faf3788f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:42.516 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.501+0000 7faf3788f640 1 -- 172.21.15.79:0/3091657876 >> v1:172.21.15.79:6790/0 conn(0x7faf300ac5a0 legacy=0x7faf30151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.501+0000 7faf3788f640 1 -- 172.21.15.79:0/3091657876 shutdown_connections 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.501+0000 7faf3788f640 1 -- 172.21.15.79:0/3091657876 >> 172.21.15.79:0/3091657876 conn(0x7faf3006b660 msgr2=0x7faf3006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.502+0000 7faf3788f640 1 -- 172.21.15.79:0/3091657876 shutdown_connections 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.503+0000 7faf3788f640 1 -- 172.21.15.79:0/3091657876 wait complete. 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.503+0000 7faf3788f640 10 monclient: build_initial_monmap 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.503+0000 7faf3788f640 10 monclient: monmap: 2024-03-29T10:49:42.517 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:42.518 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:42.518 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:42.518 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:42.518 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:42.518 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:42.518 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 1 librados: starting msgr at 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 1 librados: starting objecter 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 1 Processor -- start 2024-03-29T10:49:42.519 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 1 -- start start 2024-03-29T10:49:42.520 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 1 librados: setting wanted keys 2024-03-29T10:49:42.520 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 1 librados: calling monclient init 2024-03-29T10:49:42.520 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.504+0000 7faf3788f640 10 monclient: init 2024-03-29T10:49:42.520 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:42.520 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:42.520 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient(hunting): picked mon.noname-b con 0x7faf300ac5a0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:42.521 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient(hunting): picked mon.noname-a con 0x7faf300b0770 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:42.521 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient(hunting): picked mon.noname-c con 0x7faf3014a700 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:42.521 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30122660 con 0x7faf300b0770 2024-03-29T10:49:42.521 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf301e4510 con 0x7faf300ac5a0 2024-03-29T10:49:42.521 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30123770 con 0x7faf3014a700 2024-03-29T10:49:42.521 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:42.522 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.505+0000 7faf3788f640 10 monclient(hunting): authenticate will time out at 1101.797428s 2024-03-29T10:49:42.522 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.508+0000 7faf34e2a640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7faf300b0770 0x7faf30120270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:34541/0 (socket says 172.21.15.79:34541) 2024-03-29T10:49:42.522 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.508+0000 7faf34e2a640 1 -- 172.21.15.79:0/2194284019 learned_addr learned my addr 172.21.15.79:0/2194284019 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:42.522 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.508+0000 7faf25ffb640 10 client.?.objecter ms_handle_connect 0x7faf300b0770 2024-03-29T10:49:42.522 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.508+0000 7faf25ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:42.522 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.508+0000 7faf25ffb640 10 client.?.objecter ms_handle_connect 0x7faf3014a700 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.508+0000 7faf25ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443612429 0 0) 0x7faf30123770 con 0x7faf3014a700 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:49:42.523 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf14004eb0 con 0x7faf3014a700 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267054837 0 0) 0x7faf30122660 con 0x7faf300b0770 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.509+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf30123770 con 0x7faf300b0770 2024-03-29T10:49:42.524 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 client.?.objecter ms_handle_connect 0x7faf300ac5a0 2024-03-29T10:49:42.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:42.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637612867 0 0) 0x7faf301e4510 con 0x7faf300ac5a0 2024-03-29T10:49:42.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:42.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:42.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:42.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf30122660 con 0x7faf300ac5a0 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788264809 0 0) 0x7faf14004eb0 con 0x7faf3014a700 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf301e4510 con 0x7faf3014a700 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faf28001860 con 0x7faf3014a700 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:42.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): dump: 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:42.527 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.510+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2102402495 0 0) 0x7faf30123770 con 0x7faf300b0770 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf14004eb0 con 0x7faf300b0770 2024-03-29T10:49:42.528 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2721200557 0 0) 0x7faf301e4510 con 0x7faf3014a700 2024-03-29T10:49:42.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 >> v1:172.21.15.82:6789/0 conn(0x7faf300ac5a0 legacy=0x7faf30123060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 >> v1:172.21.15.79:6789/0 conn(0x7faf300b0770 legacy=0x7faf30120270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:42.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 1 monclient: found mon.c 2024-03-29T10:49:42.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:42.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:42.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30120980 con 0x7faf3014a700 2024-03-29T10:49:42.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:42.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:42.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:42.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:49:42.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 10 monclient: _renew_subs 2024-03-29T10:49:42.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:42.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf301e4a30 con 0x7faf3014a700 2024-03-29T10:49:42.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:42.533 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 10 monclient: _renew_subs 2024-03-29T10:49:42.533 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:42.534 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.511+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf30121520 con 0x7faf3014a700 2024-03-29T10:49:42.534 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.512+0000 7faf3788f640 1 librados: init done 2024-03-29T10:49:42.534 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf3788f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:42.534 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf28004e10 con 0x7faf3014a700 2024-03-29T10:49:42.534 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:42.535 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faf28005120 con 0x7faf3014a700 2024-03-29T10:49:42.535 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:42.535 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf25ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:42.535 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf25ffb640 10 monclient: dump: 2024-03-29T10:49:42.535 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:42.536 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.514+0000 7faf277fe640 4 set_mon_vals no callback set 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.515+0000 7faf3788f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.515+0000 7faf3788f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.515+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf300ab560 con 0x7faf3014a700 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.516+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7faf2801b0f0 con 0x7faf3014a700 2024-03-29T10:49:42.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7faf28084d90 con 0x7faf3014a700 2024-03-29T10:49:42.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf25ffb640 10 client.4304.objecter ms_dispatch 0x7faf30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:42.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf25ffb640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:42.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf25ffb640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:42.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf25ffb640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:49:42.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf3562b640 10 monclient: get_auth_request con 0x7faf1406dea0 auth_method 0 2024-03-29T10:49:42.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.520+0000 7faf25ffb640 10 client.4304.objecter ms_handle_connect 0x7faf1406dea0 2024-03-29T10:49:42.539 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.526+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf28050670 con 0x7faf3014a700 2024-03-29T10:49:42.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.527+0000 7faf25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:42.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.527+0000 7faf25ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:42.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.854+0000 7faf3788f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.854+0000 7faf3788f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.854+0000 7faf3788f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:42.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.854+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf301db0d0 con 0x7faf3014a700 2024-03-29T10:49:42.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.855+0000 7faf25ffb640 1 -- 172.21.15.79:0/2194284019 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7faf301db0d0 con 0x7faf3014a700 2024-03-29T10:49:42.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.855+0000 7faf25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.855+0000 7faf25ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:42.838 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:42.838 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:42.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.859+0000 7faf3788f640 10 librados: watch_flush enter 2024-03-29T10:49:42.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 10 librados: watch_flush exit 2024-03-29T10:49:42.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:49:42.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:49:42.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 >> v1:172.21.15.79:6816/4290694627 conn(0x7faf1406dea0 legacy=0x7faf14070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 10 monclient: shutdown 2024-03-29T10:49:42.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:42.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 >> v1:172.21.15.79:6790/0 conn(0x7faf3014a700 legacy=0x7faf30120e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 shutdown_connections 2024-03-29T10:49:42.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.862+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 >> 172.21.15.79:0/2194284019 conn(0x7faf3006b660 msgr2=0x7faf300abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:42.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.863+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 shutdown_connections 2024-03-29T10:49:42.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.864+0000 7faf3788f640 1 -- 172.21.15.79:0/2194284019 wait complete. 2024-03-29T10:49:42.848 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:42.866+0000 7faf3788f640 1 librados: shutdown 2024-03-29T10:49:42.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:49.675 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:49.826 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.844+0000 7f27163d1640 10 monclient: get_monmap_and_config 2024-03-29T10:49:49.826 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.844+0000 7f27163d1640 10 monclient: build_initial_monmap 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.844+0000 7f27163d1640 10 monclient: monmap: 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:49.845228+0000 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:49.845228+0000 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:49.827 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.845+0000 7f27163d1640 1 Processor -- start 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.845+0000 7f27163d1640 1 -- start start 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.845+0000 7f27163d1640 10 monclient: init 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient(hunting): picked mon.noname-c con 0x7f27100a9750 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient(hunting): picked mon.noname-b con 0x7f27100ae220 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient(hunting): picked mon.noname-a con 0x7f271013b9d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2710142490 con 0x7f271013b9d0 2024-03-29T10:49:49.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27100ac130 con 0x7f27100a9750 2024-03-29T10:49:49.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27100ac410 con 0x7f27100ae220 2024-03-29T10:49:49.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:49.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.846+0000 7f27163d1640 10 monclient(hunting): authenticate will time out at 1109.138810s 2024-03-29T10:49:49.834 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f2715bd0640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f271013b9d0 0x7f2710149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:52787/0 (socket says 172.21.15.79:52787) 2024-03-29T10:49:49.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f2715bd0640 1 -- 172.21.15.79:0/365220440 learned_addr learned my addr 172.21.15.79:0/365220440 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187899086 0 0) 0x7f2710142490 con 0x7f271013b9d0 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f270ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f270ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f270ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f270ffff640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.853+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f8003440 con 0x7f271013b9d0 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705297492 0 0) 0x7f27100ac410 con 0x7f27100ae220 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710142490 con 0x7f27100ae220 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815889425 0 0) 0x7f26f8003440 con 0x7f271013b9d0 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 >> v1:172.21.15.79:6790/0 conn(0x7f27100ae220 legacy=0x7f271013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 >> v1:172.21.15.82:6789/0 conn(0x7f27100a9750 legacy=0x7f2710141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 monclient: found mon.noname-a 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27100afa40 con 0x7f271013b9d0 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: _finish_auth 0 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:49.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2700001690 con 0x7f271013b9d0 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f270ffff640 10 monclient: dump: 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f27163d1640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:49:49.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.854+0000 7f27163d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27000017f0 con 0x7f271013b9d0 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f270ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f270ffff640 1 -- 172.21.15.79:0/365220440 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2700001b80 con 0x7f271013b9d0 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f270ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f270ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f270ffff640 10 monclient: dump: 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:49.842 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f27163d1640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.855+0000 7f27163d1640 4 set_mon_vals no callback set 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.856+0000 7f27163d1640 10 monclient: shutdown 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.856+0000 7f27163d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.856+0000 7f27163d1640 1 -- 172.21.15.79:0/365220440 >> v1:172.21.15.79:6789/0 conn(0x7f271013b9d0 legacy=0x7f2710149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.856+0000 7f27163d1640 1 -- 172.21.15.79:0/365220440 shutdown_connections 2024-03-29T10:49:49.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.856+0000 7f27163d1640 1 -- 172.21.15.79:0/365220440 >> 172.21.15.79:0/365220440 conn(0x7f27100673e0 msgr2=0x7f27100b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:49.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.857+0000 7f27163d1640 1 -- 172.21.15.79:0/365220440 shutdown_connections 2024-03-29T10:49:49.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.858+0000 7f27163d1640 1 -- 172.21.15.79:0/365220440 wait complete. 2024-03-29T10:49:49.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.859+0000 7f27163d1640 10 monclient: build_initial_monmap 2024-03-29T10:49:49.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.859+0000 7f27163d1640 10 monclient: monmap: 2024-03-29T10:49:49.844 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.860+0000 7f27163d1640 1 librados: starting msgr at 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.860+0000 7f27163d1640 1 librados: starting objecter 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.860+0000 7f27163d1640 1 Processor -- start 2024-03-29T10:49:49.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 1 -- start start 2024-03-29T10:49:49.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 1 librados: setting wanted keys 2024-03-29T10:49:49.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 1 librados: calling monclient init 2024-03-29T10:49:49.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 10 monclient: init 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.861+0000 7f27163d1640 10 monclient(hunting): picked mon.noname-a con 0x7f27100a9750 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 10 monclient(hunting): picked mon.noname-b con 0x7f27100ae220 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 10 monclient(hunting): picked mon.noname-c con 0x7f271013b9d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2710130360 con 0x7f27100a9750 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27101dc6d0 con 0x7f27100ae220 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27101dbaa0 con 0x7f271013b9d0 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:49.847 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.862+0000 7f27163d1640 10 monclient(hunting): authenticate will time out at 1109.154159s 2024-03-29T10:49:49.848 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.866+0000 7f2715bd0640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f271013b9d0 0x7f271012fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43825/0 (socket says 172.21.15.79:43825) 2024-03-29T10:49:49.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.867+0000 7f2715bd0640 1 -- 172.21.15.79:0/3564688189 learned_addr learned my addr 172.21.15.79:0/3564688189 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:49.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 client.?.objecter ms_handle_connect 0x7f271013b9d0 2024-03-29T10:49:49.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:49.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 client.?.objecter ms_handle_connect 0x7f27100a9750 2024-03-29T10:49:49.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213056940 0 0) 0x7f2710130360 con 0x7f27100a9750 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.870+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26fc004eb0 con 0x7f27100a9750 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2237142506 0 0) 0x7f26fc004eb0 con 0x7f27100a9750 2024-03-29T10:49:49.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2710130360 con 0x7f27100a9750 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2708001cb0 con 0x7f27100a9750 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient(hunting): dump: 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:49.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1494169394 0 0) 0x7f2710130360 con 0x7f27100a9750 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 >> v1:172.21.15.79:6790/0 conn(0x7f271013b9d0 legacy=0x7f271012fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 >> v1:172.21.15.82:6789/0 conn(0x7f27100ae220 legacy=0x7f271012f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 monclient: found mon.a 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27100724d0 con 0x7f27100a9750 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f270dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f27163d1640 5 monclient: authenticate success, global_id 4296 2024-03-29T10:49:49.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f27163d1640 10 monclient: _renew_subs 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f27163d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.871+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2710072720 con 0x7f27100a9750 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.872+0000 7f27163d1640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.872+0000 7f27163d1640 10 monclient: _renew_subs 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.872+0000 7f27163d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.872+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2710071b10 con 0x7f27100a9750 2024-03-29T10:49:49.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.872+0000 7f27163d1640 1 librados: init done 2024-03-29T10:49:49.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.873+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2708003990 con 0x7f27100a9750 2024-03-29T10:49:49.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.873+0000 7f270dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:49.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.873+0000 7f270effd640 4 set_mon_vals no callback set 2024-03-29T10:49:49.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.874+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2708003d60 con 0x7f27100a9750 2024-03-29T10:49:49.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.874+0000 7f270dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.874+0000 7f270dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.874+0000 7f270dffb640 10 monclient: dump: 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:49.859 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:49.860 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:49.860 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.876+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f2708003ec0 con 0x7f27100a9750 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f2714bce640 10 monclient: get_auth_request con 0x7f26fc06de00 auth_method 0 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f2708083a10 con 0x7f27100a9750 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f270dffb640 10 client.4296.objecter ms_dispatch 0x7f2710000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f270dffb640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f270dffb640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f270dffb640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T10:49:49.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.877+0000 7f27163d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.878+0000 7f27163d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.878+0000 7f27163d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:49:49.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.878+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f0005080 con 0x7f27100a9750 2024-03-29T10:49:49.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.878+0000 7f270dffb640 10 client.4296.objecter ms_handle_connect 0x7f26fc06de00 2024-03-29T10:49:49.868 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.886+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f270804f350 con 0x7f27100a9750 2024-03-29T10:49:49.868 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.886+0000 7f270dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.868 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:49.886+0000 7f270dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:50.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.221+0000 7f27163d1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:50.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.221+0000 7f27163d1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:50.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.221+0000 7f27163d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:49:50.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.221+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26f0005330 con 0x7f27100a9750 2024-03-29T10:49:50.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.224+0000 7f270dffb640 1 -- 172.21.15.79:0/3564688189 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f27080830c0 con 0x7f27100a9750 2024-03-29T10:49:50.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.224+0000 7f270dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:50.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.224+0000 7f270dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:50.207 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:50.207 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:50.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.228+0000 7f27163d1640 10 librados: watch_flush enter 2024-03-29T10:49:50.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.228+0000 7f27163d1640 10 librados: watch_flush exit 2024-03-29T10:49:50.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.228+0000 7f27163d1640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T10:49:50.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.228+0000 7f27163d1640 10 client.4296.objecter successfully canceled tick 2024-03-29T10:49:50.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.231+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 >> v1:172.21.15.79:6816/4290694627 conn(0x7f26fc06de00 legacy=0x7f26fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:50.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.231+0000 7f27163d1640 10 monclient: shutdown 2024-03-29T10:49:50.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.231+0000 7f27163d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:50.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.231+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 >> v1:172.21.15.79:6789/0 conn(0x7f27100a9750 legacy=0x7f27101dbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:50.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.232+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 shutdown_connections 2024-03-29T10:49:50.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.232+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 >> 172.21.15.79:0/3564688189 conn(0x7f27100673e0 msgr2=0x7f2710069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:50.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.233+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 shutdown_connections 2024-03-29T10:49:50.217 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.235+0000 7f27163d1640 1 -- 172.21.15.79:0/3564688189 wait complete. 2024-03-29T10:49:50.217 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:50.235+0000 7f27163d1640 1 librados: shutdown 2024-03-29T10:49:50.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:57.041 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.205+0000 7ff47dbfb640 10 monclient: get_monmap_and_config 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.205+0000 7ff47dbfb640 10 monclient: build_initial_monmap 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.205+0000 7ff47dbfb640 10 monclient: monmap: 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:49:57.207146+0000 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:49:57.207146+0000 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:57.188 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:57.189 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:49:57.189 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:49:57.189 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:57.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.206+0000 7ff47dbfb640 1 Processor -- start 2024-03-29T10:49:57.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.206+0000 7ff47dbfb640 1 -- start start 2024-03-29T10:49:57.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.206+0000 7ff47dbfb640 10 monclient: init 2024-03-29T10:49:57.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.207+0000 7ff47dbfb640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:57.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.207+0000 7ff47dbfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:57.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.208+0000 7ff47dbfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff47806ebd0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:57.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.208+0000 7ff47dbfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff47806efb0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:57.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.209+0000 7ff4777fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7ff47806ebd0 0x7ff47813b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:52148/0 (socket says 172.21.15.79:52148) 2024-03-29T10:49:57.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.211+0000 7ff4777fe640 1 -- 172.21.15.79:0/4415785 learned_addr learned my addr 172.21.15.79:0/4415785 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:57.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.211+0000 7ff47dbfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff47813ed90 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:57.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.212+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff478070b40 con 0x7ff47813ed90 2024-03-29T10:49:57.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.212+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff478070e20 con 0x7ff47806efb0 2024-03-29T10:49:57.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.212+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff478071100 con 0x7ff47806ebd0 2024-03-29T10:49:57.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.212+0000 7ff47dbfb640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:57.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.212+0000 7ff47dbfb640 10 monclient(hunting): authenticate will time out at 1116.505633s 2024-03-29T10:49:57.198 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832891306 0 0) 0x7ff478071100 con 0x7ff47806ebd0 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff460003520 con 0x7ff47806ebd0 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496477286 0 0) 0x7ff478070e20 con 0x7ff47806efb0 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff478071100 con 0x7ff47806efb0 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542447843 0 0) 0x7ff478070b40 con 0x7ff47813ed90 2024-03-29T10:49:57.199 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.213+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff478070e20 con 0x7ff47813ed90 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3985109331 0 0) 0x7ff460003520 con 0x7ff47806ebd0 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 >> v1:172.21.15.82:6789/0 conn(0x7ff47806efb0 legacy=0x7ff47813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 >> v1:172.21.15.79:6789/0 conn(0x7ff47813ed90 legacy=0x7ff478149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 monclient: found mon.noname-b 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47813c040 con 0x7ff47806ebd0 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:57.200 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff46c001690 con 0x7ff47806ebd0 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: dump: 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:57.201 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff46c0018b0 con 0x7ff47806ebd0 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 1 -- 172.21.15.79:0/4415785 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff46c001c40 con 0x7ff47806ebd0 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff4767fc640 10 monclient: dump: 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:57.202 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff47dbfb640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff47dbfb640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.214+0000 7ff47dbfb640 4 set_mon_vals no callback set 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.215+0000 7ff47dbfb640 10 monclient: shutdown 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.215+0000 7ff47dbfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.215+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 >> v1:172.21.15.79:6790/0 conn(0x7ff47806ebd0 legacy=0x7ff47813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.216+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 shutdown_connections 2024-03-29T10:49:57.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.216+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 >> 172.21.15.79:0/4415785 conn(0x7ff4780b4200 msgr2=0x7ff4780b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.218+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 shutdown_connections 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.218+0000 7ff47dbfb640 1 -- 172.21.15.79:0/4415785 wait complete. 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.219+0000 7ff47dbfb640 10 monclient: build_initial_monmap 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.219+0000 7ff47dbfb640 10 monclient: monmap: 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:49:57.204 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.219+0000 7ff47dbfb640 1 librados: starting msgr at 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.219+0000 7ff47dbfb640 1 librados: starting objecter 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.220+0000 7ff47dbfb640 1 Processor -- start 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.222+0000 7ff47dbfb640 1 -- start start 2024-03-29T10:49:57.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.222+0000 7ff47dbfb640 1 librados: setting wanted keys 2024-03-29T10:49:57.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.222+0000 7ff47dbfb640 1 librados: calling monclient init 2024-03-29T10:49:57.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.222+0000 7ff47dbfb640 10 monclient: init 2024-03-29T10:49:57.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:57.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff47806ebd0 addr v1:172.21.15.82:6789/0 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff47806efb0 addr v1:172.21.15.79:6790/0 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff47813ed90 addr v1:172.21.15.79:6789/0 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4781c9400 con 0x7ff47813ed90 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4781c8120 con 0x7ff47806ebd0 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4781c8400 con 0x7ff47806efb0 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:57.208 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.223+0000 7ff47dbfb640 10 monclient(hunting): authenticate will time out at 1116.516812s 2024-03-29T10:49:57.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.225+0000 7ff476ffd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7ff47806efb0 0x7ff4781db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:36506/0 (socket says 172.21.15.79:36506) 2024-03-29T10:49:57.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.226+0000 7ff476ffd640 1 -- 172.21.15.79:0/3966877097 learned_addr learned my addr 172.21.15.79:0/3966877097 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:49:57.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.226+0000 7ff45ffff640 10 client.?.objecter ms_handle_connect 0x7ff47806efb0 2024-03-29T10:49:57.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.226+0000 7ff45ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:57.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563148203 0 0) 0x7ff4781c8400 con 0x7ff47806efb0 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff464004eb0 con 0x7ff47806efb0 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 client.?.objecter ms_handle_connect 0x7ff47806ebd0 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 900068238 0 0) 0x7ff464004eb0 con 0x7ff47806efb0 2024-03-29T10:49:57.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4781c8400 con 0x7ff47806efb0 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff4680019c0 con 0x7ff47806efb0 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): dump: 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:57.213 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.227+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228876597 0 0) 0x7ff4781c8120 con 0x7ff47806ebd0 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff464004eb0 con 0x7ff47806ebd0 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 333547756 0 0) 0x7ff4781c8400 con 0x7ff47806efb0 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 >> v1:172.21.15.82:6789/0 conn(0x7ff47806ebd0 legacy=0x7ff4781c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.228+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 >> v1:172.21.15.79:6789/0 conn(0x7ff47813ed90 legacy=0x7ff4781dc070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:57.218 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:57.218 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 1 monclient: found mon.c 2024-03-29T10:49:57.218 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:57.218 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4781c86e0 con 0x7ff47806efb0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 10 monclient: _finish_auth 0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff45ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 10 monclient: _renew_subs 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4781c8ee0 con 0x7ff47806efb0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 10 monclient: _renew_subs 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4781c8b00 con 0x7ff47806efb0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.232+0000 7ff47dbfb640 1 librados: init done 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff468001c80 con 0x7ff47806efb0 2024-03-29T10:49:57.219 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff45ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff468003a40 con 0x7ff47806efb0 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff45ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff45ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff45ffff640 10 monclient: dump: 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:49:57.220 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.233+0000 7ff4757fa640 4 set_mon_vals no callback set 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.234+0000 7ff47dbfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.235+0000 7ff47dbfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.236+0000 7ff47dbfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.236+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff458005080 con 0x7ff47806efb0 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.237+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7ff468003ba0 con 0x7ff47806efb0 2024-03-29T10:49:57.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.238+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7ff468084ae0 con 0x7ff47806efb0 2024-03-29T10:49:57.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.238+0000 7ff45ffff640 10 client.4313.objecter ms_dispatch 0x7ff478000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:57.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.238+0000 7ff45ffff640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:57.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.238+0000 7ff45ffff640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:57.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.238+0000 7ff4777fe640 10 monclient: get_auth_request con 0x7ff46406ddd0 auth_method 0 2024-03-29T10:49:57.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.238+0000 7ff45ffff640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T10:49:57.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.239+0000 7ff45ffff640 10 client.4313.objecter ms_handle_connect 0x7ff46406ddd0 2024-03-29T10:49:57.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.245+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4680503c0 con 0x7ff47806efb0 2024-03-29T10:49:57.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.245+0000 7ff45ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:57.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.245+0000 7ff45ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:57.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.565+0000 7ff47dbfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:57.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.565+0000 7ff47dbfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:57.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.565+0000 7ff47dbfb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:49:57.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.565+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff458005330 con 0x7ff47806efb0 2024-03-29T10:49:57.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.566+0000 7ff45ffff640 1 -- 172.21.15.79:0/3966877097 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7ff4680522f0 con 0x7ff47806efb0 2024-03-29T10:49:57.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.566+0000 7ff45ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:57.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.566+0000 7ff45ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:57.551 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:49:57.551 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.571+0000 7ff47dbfb640 10 librados: watch_flush enter 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.571+0000 7ff47dbfb640 10 librados: watch_flush exit 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 10 client.4313.objecter successfully canceled tick 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 >> v1:172.21.15.79:6816/4290694627 conn(0x7ff46406ddd0 legacy=0x7ff464070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 10 monclient: shutdown 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:57.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 >> v1:172.21.15.79:6790/0 conn(0x7ff47806efb0 legacy=0x7ff4781db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:57.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 shutdown_connections 2024-03-29T10:49:57.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.572+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 >> 172.21.15.79:0/3966877097 conn(0x7ff4780b4200 msgr2=0x7ff4780b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:57.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.573+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 shutdown_connections 2024-03-29T10:49:57.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.573+0000 7ff47dbfb640 1 -- 172.21.15.79:0/3966877097 wait complete. 2024-03-29T10:49:57.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:49:57.575+0000 7ff47dbfb640 1 librados: shutdown 2024-03-29T10:49:57.575 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:04.382 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:04.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.555+0000 7f4b13c43640 10 monclient: get_monmap_and_config 2024-03-29T10:50:04.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.555+0000 7f4b13c43640 10 monclient: build_initial_monmap 2024-03-29T10:50:04.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.555+0000 7f4b13c43640 10 monclient: monmap: 2024-03-29T10:50:04.537 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:04.556587+0000 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:04.556587+0000 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:04.538 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.543 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.562+0000 7f4b13c43640 1 Processor -- start 2024-03-29T10:50:04.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.562+0000 7f4b13c43640 1 -- start start 2024-03-29T10:50:04.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.562+0000 7f4b13c43640 10 monclient: init 2024-03-29T10:50:04.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:04.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:04.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient(hunting): picked mon.noname-c con 0x7f4b0c1457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0c13f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0c0a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c0aaef0 con 0x7f4b0c13f650 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c145bd0 con 0x7f4b0c1457f0 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c145eb0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.563+0000 7f4b13c43640 10 monclient(hunting): authenticate will time out at 1123.855547s 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.567+0000 7f4b119df640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f4b0c1457f0 0x7f4b0c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:36251/0 (socket says 172.21.15.79:36251) 2024-03-29T10:50:04.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.567+0000 7f4b119df640 1 -- 172.21.15.79:0/1623096383 learned_addr learned my addr 172.21.15.79:0/1623096383 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:04.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458463070 0 0) 0x7f4b0c145bd0 con 0x7f4b0c1457f0 2024-03-29T10:50:04.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af8003520 con 0x7f4b0c1457f0 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169395948 0 0) 0x7f4b0c0aaef0 con 0x7f4b0c13f650 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c145bd0 con 0x7f4b0c13f650 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104436594 0 0) 0x7f4af8003520 con 0x7f4b0c1457f0 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 >> v1:172.21.15.79:6790/0 conn(0x7f4b0c0a9d70 legacy=0x7f4b0c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 >> v1:172.21.15.79:6789/0 conn(0x7f4b0c13f650 legacy=0x7f4b0c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 monclient: found mon.noname-c 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c146340 con 0x7f4b0c1457f0 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: _finish_auth 0 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4b04001a10 con 0x7f4b0c1457f0 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b13c43640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b13c43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.572+0000 7f4b109dd640 10 monclient: dump: 2024-03-29T10:50:04.556 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b13c43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b04001b70 con 0x7f4b0c1457f0 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b109dd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b109dd640 1 -- 172.21.15.79:0/1623096383 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4b04001f80 con 0x7f4b0c1457f0 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b109dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b109dd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b109dd640 10 monclient: dump: 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b13c43640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:04.558 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.573+0000 7f4b13c43640 4 set_mon_vals no callback set 2024-03-29T10:50:04.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.574+0000 7f4b13c43640 10 monclient: shutdown 2024-03-29T10:50:04.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.574+0000 7f4b13c43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:04.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.574+0000 7f4b13c43640 1 -- 172.21.15.79:0/1623096383 >> v1:172.21.15.82:6789/0 conn(0x7f4b0c1457f0 legacy=0x7f4b0c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.576+0000 7f4b13c43640 1 -- 172.21.15.79:0/1623096383 shutdown_connections 2024-03-29T10:50:04.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.576+0000 7f4b13c43640 1 -- 172.21.15.79:0/1623096383 >> 172.21.15.79:0/1623096383 conn(0x7f4b0c066a00 msgr2=0x7f4b0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:04.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.579+0000 7f4b13c43640 1 -- 172.21.15.79:0/1623096383 shutdown_connections 2024-03-29T10:50:04.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.579+0000 7f4b13c43640 1 -- 172.21.15.79:0/1623096383 wait complete. 2024-03-29T10:50:04.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.580+0000 7f4b13c43640 10 monclient: build_initial_monmap 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.580+0000 7f4b13c43640 10 monclient: monmap: 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.580+0000 7f4b13c43640 1 librados: starting msgr at 2024-03-29T10:50:04.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.580+0000 7f4b13c43640 1 librados: starting objecter 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.581+0000 7f4b13c43640 1 Processor -- start 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.581+0000 7f4b13c43640 1 -- start start 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.581+0000 7f4b13c43640 1 librados: setting wanted keys 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.581+0000 7f4b13c43640 1 librados: calling monclient init 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.581+0000 7f4b13c43640 10 monclient: init 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient(hunting): picked mon.noname-b con 0x7f4b0c0a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient(hunting): picked mon.noname-c con 0x7f4b0c13f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient(hunting): picked mon.noname-a con 0x7f4b0c1457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c135ff0 con 0x7f4b0c1457f0 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c144b70 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c144de0 con 0x7f4b0c13f650 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:04.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.582+0000 7f4b13c43640 10 monclient(hunting): authenticate will time out at 1123.874342s 2024-03-29T10:50:04.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.587+0000 7f4b121e0640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f4b0c1457f0 0x7f4b0c1de8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:41571/0 (socket says 172.21.15.79:41571) 2024-03-29T10:50:04.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.587+0000 7f4b121e0640 1 -- 172.21.15.79:0/4207048602 learned_addr learned my addr 172.21.15.79:0/4207048602 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:04.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.587+0000 7f4b027fc640 10 client.?.objecter ms_handle_connect 0x7f4b0c1457f0 2024-03-29T10:50:04.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.587+0000 7f4b027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.587+0000 7f4b027fc640 10 client.?.objecter ms_handle_connect 0x7f4b0c0a9d70 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.587+0000 7f4b027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808282516 0 0) 0x7f4b0c144b70 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af0004eb0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537375162 0 0) 0x7f4b0c135ff0 con 0x7f4b0c1457f0 2024-03-29T10:50:04.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c144b70 con 0x7f4b0c1457f0 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 client.?.objecter ms_handle_connect 0x7f4b0c13f650 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.588+0000 7f4b027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592942695 0 0) 0x7f4b0c144de0 con 0x7f4b0c13f650 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c135ff0 con 0x7f4b0c13f650 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214703897 0 0) 0x7f4af0004eb0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.580 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b0c144de0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4b04001e80 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): dump: 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.581 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4018111346 0 0) 0x7f4b0c135ff0 con 0x7f4b0c13f650 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af0004eb0 con 0x7f4b0c13f650 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4afc0018f0 con 0x7f4b0c13f650 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient(hunting): dump: 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:04.582 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236964149 0 0) 0x7f4b0c144de0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 >> v1:172.21.15.79:6790/0 conn(0x7f4b0c13f650 legacy=0x7f4b0c1d6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 >> v1:172.21.15.79:6789/0 conn(0x7f4b0c1457f0 legacy=0x7f4b0c1de8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 monclient: found mon.b 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c1450c0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 5 monclient: authenticate success, global_id 4294 2024-03-29T10:50:04.583 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 10 monclient: _renew_subs 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c13de90 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 10 monclient: _renew_subs 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c135940 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.589+0000 7f4b13c43640 1 librados: init done 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b040020e0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b03fff640 4 set_mon_vals no callback set 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4b04004fb0 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.590+0000 7f4b027fc640 10 monclient: dump: 2024-03-29T10:50:04.584 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.592+0000 7f4b13c43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.592+0000 7f4b13c43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.592+0000 7f4b13c43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.592+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af4005080 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.594+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f4b0401af80 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.595+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f4b04084d80 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.595+0000 7f4b111de640 10 monclient: get_auth_request con 0x7f4af006df90 auth_method 0 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.596+0000 7f4b027fc640 10 client.4294.objecter ms_dispatch 0x7f4b0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.596+0000 7f4b027fc640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.596+0000 7f4b027fc640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.596+0000 7f4b027fc640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.596+0000 7f4b027fc640 10 client.4294.objecter ms_handle_connect 0x7f4af006df90 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.601+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b04050660 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.601+0000 7f4b027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:04.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.601+0000 7f4b027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:04.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.947+0000 7f4b13c43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.947+0000 7f4b13c43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.947+0000 7f4b13c43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:04.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.947+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4af4005600 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.952+0000 7f4b027fc640 1 -- 172.21.15.79:0/4207048602 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f4b04052590 con 0x7f4b0c0a9d70 2024-03-29T10:50:04.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.952+0000 7f4b027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.952+0000 7f4b027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:04.935 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:04.935 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:04.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.957+0000 7f4b13c43640 10 librados: watch_flush enter 2024-03-29T10:50:04.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.957+0000 7f4b13c43640 10 librados: watch_flush exit 2024-03-29T10:50:04.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.957+0000 7f4b13c43640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T10:50:04.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.957+0000 7f4b13c43640 10 client.4294.objecter successfully canceled tick 2024-03-29T10:50:04.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.959+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 >> v1:172.21.15.79:6816/4290694627 conn(0x7f4af006df90 legacy=0x7f4af0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.960+0000 7f4b13c43640 10 monclient: shutdown 2024-03-29T10:50:04.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.960+0000 7f4b13c43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:04.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.960+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 >> v1:172.21.15.82:6789/0 conn(0x7f4b0c0a9d70 legacy=0x7f4b0c1d57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.960+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 shutdown_connections 2024-03-29T10:50:04.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.960+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 >> 172.21.15.79:0/4207048602 conn(0x7f4b0c066a00 msgr2=0x7f4b0c136270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:04.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.961+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 shutdown_connections 2024-03-29T10:50:04.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.961+0000 7f4b13c43640 1 -- 172.21.15.79:0/4207048602 wait complete. 2024-03-29T10:50:04.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:04.964+0000 7f4b13c43640 1 librados: shutdown 2024-03-29T10:50:04.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:11.771 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:11.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.936+0000 7f00ffe4e640 10 monclient: get_monmap_and_config 2024-03-29T10:50:11.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.936+0000 7f00ffe4e640 10 monclient: build_initial_monmap 2024-03-29T10:50:11.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.936+0000 7f00ffe4e640 10 monclient: monmap: 2024-03-29T10:50:11.918 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:11.937553+0000 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:11.937553+0000 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:11.919 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.937+0000 7f00ffe4e640 1 Processor -- start 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.937+0000 7f00ffe4e640 1 -- start start 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.937+0000 7f00ffe4e640 10 monclient: init 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.938+0000 7f00ffe4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.938+0000 7f00ffe4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 10 monclient(hunting): picked mon.noname-b con 0x7f00f81422b0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 10 monclient(hunting): picked mon.noname-c con 0x7f00f8142690 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:11.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fdbea640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f00f81422b0 0x7f00f813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35050/0 (socket says 172.21.15.79:35050) 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 10 monclient(hunting): picked mon.noname-a con 0x7f00f80b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fd3e9640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f00f8142690 0x7f00f80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:59944/0 (socket says 172.21.15.79:59944) 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fd3e9640 1 -- 172.21.15.79:0/3420010852 learned_addr learned my addr 172.21.15.79:0/3420010852 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f8142c80 con 0x7f00f80b17d0 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f80aae40 con 0x7f00f8142690 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f813fbd0 con 0x7f00f81422b0 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00ffe4e640 10 monclient(hunting): authenticate will time out at 1131.231942s 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8188905 0 0) 0x7f00f80aae40 con 0x7f00f8142690 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fcbe8640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fcbe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fcbe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.939+0000 7f00fcbe8640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:50:11.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00e4003520 con 0x7f00f8142690 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937084123 0 0) 0x7f00e4003520 con 0x7f00f8142690 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 >> v1:172.21.15.79:6790/0 conn(0x7f00f81422b0 legacy=0x7f00f813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 >> v1:172.21.15.79:6789/0 conn(0x7f00f80b17d0 legacy=0x7f00f80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: _finish_hunting 0 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 monclient: found mon.noname-c 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8150930 con 0x7f00f8142690 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: _finish_auth 0 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: _check_auth_tickets 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f00e8001690 con 0x7f00f8142690 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00fcbe8640 10 monclient: dump: 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00ffe4e640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.940+0000 7f00ffe4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00e80017f0 con 0x7f00f8142690 2024-03-29T10:50:11.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00fcbe8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00fcbe8640 1 -- 172.21.15.79:0/3420010852 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f00e8001bc0 con 0x7f00f8142690 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00fcbe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00fcbe8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00fcbe8640 10 monclient: dump: 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00ffe4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.941+0000 7f00ffe4e640 4 set_mon_vals no callback set 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 10 monclient: shutdown 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 >> v1:172.21.15.82:6789/0 conn(0x7f00f8142690 legacy=0x7f00f80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 shutdown_connections 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 >> 172.21.15.79:0/3420010852 conn(0x7f00f8066a00 msgr2=0x7f00f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 shutdown_connections 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.942+0000 7f00ffe4e640 1 -- 172.21.15.79:0/3420010852 wait complete. 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.943+0000 7f00ffe4e640 10 monclient: build_initial_monmap 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 10 monclient: monmap: 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:11.938 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 1 librados: starting msgr at 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 1 librados: starting objecter 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 1 Processor -- start 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 1 -- start start 2024-03-29T10:50:11.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 1 librados: setting wanted keys 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 1 librados: calling monclient init 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.944+0000 7f00ffe4e640 10 monclient: init 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient(hunting): picked mon.noname-c con 0x7f00f80b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00fdbea640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f00f80b17d0 0x7f00f80ac840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35286/0 (socket says 172.21.15.79:35286) 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00fdbea640 1 -- 172.21.15.79:0/1458655771 learned_addr learned my addr 172.21.15.79:0/1458655771 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient(hunting): picked mon.noname-a con 0x7f00f81422b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient(hunting): picked mon.noname-b con 0x7f00f8142690 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ee7fc640 10 client.?.objecter ms_handle_connect 0x7f00f80b17d0 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f80ad0a0 con 0x7f00f81422b0 2024-03-29T10:50:11.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f80ae350 con 0x7f00f8142690 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f80ae630 con 0x7f00f80b17d0 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.945+0000 7f00ffe4e640 10 monclient(hunting): authenticate will time out at 1131.238026s 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348071835 0 0) 0x7f00f80ae630 con 0x7f00f80b17d0 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00dc004eb0 con 0x7f00f80b17d0 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 client.?.objecter ms_handle_connect 0x7f00f81422b0 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 client.?.objecter ms_handle_connect 0x7f00f8142690 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853149754 0 0) 0x7f00f80ad0a0 con 0x7f00f81422b0 2024-03-29T10:50:11.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f80ae630 con 0x7f00f81422b0 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2558625845 0 0) 0x7f00dc004eb0 con 0x7f00f80b17d0 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f80ad0a0 con 0x7f00f80b17d0 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f00f4001970 con 0x7f00f80b17d0 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.946+0000 7f00ee7fc640 10 monclient(hunting): dump: 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.942 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1363253611 0 0) 0x7f00f80ae630 con 0x7f00f81422b0 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00dc004eb0 con 0x7f00f81422b0 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f00e8001b00 con 0x7f00f81422b0 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient(hunting): dump: 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:11.943 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 436390690 0 0) 0x7f00f80ad0a0 con 0x7f00f80b17d0 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 >> v1:172.21.15.82:6789/0 conn(0x7f00f8142690 legacy=0x7f00f81e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 >> v1:172.21.15.79:6789/0 conn(0x7f00f81422b0 legacy=0x7f00f81320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 monclient: found mon.c 2024-03-29T10:50:11.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f80ad530 con 0x7f00f80b17d0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ffe4e640 5 monclient: authenticate success, global_id 4322 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ffe4e640 10 monclient: _renew_subs 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ffe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f81327d0 con 0x7f00f80b17d0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.947+0000 7f00ffe4e640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.948+0000 7f00ffe4e640 10 monclient: _renew_subs 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.948+0000 7f00ffe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.948+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f80ac1b0 con 0x7f00f80b17d0 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.948+0000 7f00ffe4e640 1 librados: init done 2024-03-29T10:50:11.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.948+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00f40047e0 con 0x7f00f80b17d0 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.949+0000 7f00ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.949+0000 7f00effff640 4 set_mon_vals no callback set 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.950+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f00f4004af0 con 0x7f00f80b17d0 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.950+0000 7f00ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.950+0000 7f00ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.950+0000 7f00ee7fc640 10 monclient: dump: 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:11.946 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.951+0000 7f00ffe4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.951+0000 7f00ffe4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.951+0000 7f00ffe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:11.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.951+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e0005080 con 0x7f00f80b17d0 2024-03-29T10:50:11.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.952+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f00f401ab00 con 0x7f00f80b17d0 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f00f4084a70 con 0x7f00f80b17d0 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00ee7fc640 10 client.4322.objecter ms_dispatch 0x7f00f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00ee7fc640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00ee7fc640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00fd3e9640 10 monclient: get_auth_request con 0x7f00dc06dee0 auth_method 0 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00ee7fc640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.953+0000 7f00ee7fc640 10 client.4322.objecter ms_handle_connect 0x7f00dc06dee0 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.959+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00f4050350 con 0x7f00f80b17d0 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.959+0000 7f00ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:11.959+0000 7f00ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:12.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.344+0000 7f00ffe4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:12.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.344+0000 7f00ffe4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:12.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.344+0000 7f00ffe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:12.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.344+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00e0005330 con 0x7f00f80b17d0 2024-03-29T10:50:12.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.345+0000 7f00ee7fc640 1 -- 172.21.15.79:0/1458655771 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f00f4052280 con 0x7f00f80b17d0 2024-03-29T10:50:12.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.345+0000 7f00ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:12.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.345+0000 7f00ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:12.328 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:12.329 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:12.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 10 librados: watch_flush enter 2024-03-29T10:50:12.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 10 librados: watch_flush exit 2024-03-29T10:50:12.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T10:50:12.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 10 client.4322.objecter successfully canceled tick 2024-03-29T10:50:12.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 >> v1:172.21.15.79:6816/4290694627 conn(0x7f00dc06dee0 legacy=0x7f00dc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:12.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 10 monclient: shutdown 2024-03-29T10:50:12.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:12.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.349+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 >> v1:172.21.15.79:6790/0 conn(0x7f00f80b17d0 legacy=0x7f00f80ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:12.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.350+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 shutdown_connections 2024-03-29T10:50:12.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.350+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 >> 172.21.15.79:0/1458655771 conn(0x7f00f8066a00 msgr2=0x7f00f80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:12.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.351+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 shutdown_connections 2024-03-29T10:50:12.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.351+0000 7f00ffe4e640 1 -- 172.21.15.79:0/1458655771 wait complete. 2024-03-29T10:50:12.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:12.351+0000 7f00ffe4e640 1 librados: shutdown 2024-03-29T10:50:12.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:19.154 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.301+0000 7f9f9958f640 10 monclient: get_monmap_and_config 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.301+0000 7f9f9958f640 10 monclient: build_initial_monmap 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.301+0000 7f9f9958f640 10 monclient: monmap: 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:19.302351+0000 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:19.302351+0000 2024-03-29T10:50:19.283 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:19.284 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:19.284 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:19.284 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:19.284 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:19.284 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.302+0000 7f9f9958f640 1 Processor -- start 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.302+0000 7f9f9958f640 1 -- start start 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.302+0000 7f9f9958f640 10 monclient: init 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient(hunting): picked mon.noname-b con 0x7f9f941457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient(hunting): picked mon.noname-a con 0x7f9f9413f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:19.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient(hunting): picked mon.noname-c con 0x7f9f940a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f940aaef0 con 0x7f9f9413f650 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f94145bd0 con 0x7f9f940a9d70 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f94145eb0 con 0x7f9f941457f0 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f9958f640 10 monclient(hunting): authenticate will time out at 1138.595591s 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f92ffd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f9f941457f0 0x7f9f9413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60282/0 (socket says 172.21.15.79:60282) 2024-03-29T10:50:19.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f92ffd640 1 -- 172.21.15.79:0/3954160930 learned_addr learned my addr 172.21.15.79:0/3954160930 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.303+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083676504 0 0) 0x7f9f94145eb0 con 0x7f9f941457f0 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f80003520 con 0x7f9f941457f0 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3100645705 0 0) 0x7f9f80003520 con 0x7f9f941457f0 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 >> v1:172.21.15.82:6789/0 conn(0x7f9f940a9d70 legacy=0x7f9f9414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.304+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 >> v1:172.21.15.79:6789/0 conn(0x7f9f9413f650 legacy=0x7f9f940b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 1 monclient: found mon.noname-b 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f94146340 con 0x7f9f941457f0 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9f8c001a10 con 0x7f9f941457f0 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f9958f640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f9958f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:19.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f91ffb640 10 monclient: dump: 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.305+0000 7f9f9958f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:19.291 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f8c001ce0 con 0x7f9f941457f0 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f9958f640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f9958f640 4 set_mon_vals no callback set 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f91ffb640 1 -- 172.21.15.79:0/3954160930 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9f8c002070 con 0x7f9f941457f0 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.306+0000 7f9f91ffb640 10 monclient: dump: 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:19.292 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 10 monclient: shutdown 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 1 -- 172.21.15.79:0/3954160930 >> v1:172.21.15.79:6790/0 conn(0x7f9f941457f0 legacy=0x7f9f9413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 1 -- 172.21.15.79:0/3954160930 shutdown_connections 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 1 -- 172.21.15.79:0/3954160930 >> 172.21.15.79:0/3954160930 conn(0x7f9f94066a00 msgr2=0x7f9f94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:19.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 1 -- 172.21.15.79:0/3954160930 shutdown_connections 2024-03-29T10:50:19.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.307+0000 7f9f9958f640 1 -- 172.21.15.79:0/3954160930 wait complete. 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.308+0000 7f9f9958f640 10 monclient: build_initial_monmap 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.308+0000 7f9f9958f640 10 monclient: monmap: 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:19.299 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.308+0000 7f9f9958f640 1 librados: starting msgr at 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.308+0000 7f9f9958f640 1 librados: starting objecter 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.309+0000 7f9f9958f640 1 Processor -- start 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.309+0000 7f9f9958f640 1 -- start start 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.309+0000 7f9f9958f640 1 librados: setting wanted keys 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.309+0000 7f9f9958f640 1 librados: calling monclient init 2024-03-29T10:50:19.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.309+0000 7f9f9958f640 10 monclient: init 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient(hunting): picked mon.noname-b con 0x7f9f940a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient(hunting): picked mon.noname-a con 0x7f9f9413f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient(hunting): picked mon.noname-c con 0x7f9f941457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9406fc50 con 0x7f9f9413f650 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f94070a40 con 0x7f9f940a9d70 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9413a040 con 0x7f9f941457f0 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f9958f640 10 monclient(hunting): authenticate will time out at 1138.602387s 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f92ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f9f940a9d70 0x7f9f941db6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:45494/0 (socket says 172.21.15.79:45494) 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.310+0000 7f9f92ffd640 1 -- 172.21.15.79:0/551573154 learned_addr learned my addr 172.21.15.79:0/551573154 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 client.?.objecter ms_handle_connect 0x7f9f940a9d70 2024-03-29T10:50:19.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616159270 0 0) 0x7f9f94070a40 con 0x7f9f940a9d70 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74004eb0 con 0x7f9f940a9d70 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 client.?.objecter ms_handle_connect 0x7f9f941457f0 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128318297 0 0) 0x7f9f74004eb0 con 0x7f9f940a9d70 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f94070a40 con 0x7f9f940a9d70 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9f8c001a90 con 0x7f9f940a9d70 2024-03-29T10:50:19.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): dump: 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.311+0000 7f9f8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693630334 0 0) 0x7f9f9413a040 con 0x7f9f941457f0 2024-03-29T10:50:19.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74004eb0 con 0x7f9f941457f0 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549453879 0 0) 0x7f9f94070a40 con 0x7f9f940a9d70 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 >> v1:172.21.15.79:6790/0 conn(0x7f9f941457f0 legacy=0x7f9f94139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 >> v1:172.21.15.79:6789/0 conn(0x7f9f9413f650 legacy=0x7f9f94139220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 monclient: found mon.b 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f94070cb0 con 0x7f9f940a9d70 2024-03-29T10:50:19.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f9958f640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f9958f640 10 monclient: _renew_subs 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f9958f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.312+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f94070f90 con 0x7f9f940a9d70 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.313+0000 7f9f9958f640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.313+0000 7f9f9958f640 10 monclient: _renew_subs 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.313+0000 7f9f9958f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.313+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f94070090 con 0x7f9f940a9d70 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.313+0000 7f9f9958f640 1 librados: init done 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f8c004e10 con 0x7f9f940a9d70 2024-03-29T10:50:19.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9f8c0051a0 con 0x7f9f940a9d70 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f917fa640 4 set_mon_vals no callback set 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.314+0000 7f9f8b7fe640 10 monclient: dump: 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:19.306 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.315+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f9f8c0053c0 con 0x7f9f940a9d70 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.315+0000 7f9f9958f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.315+0000 7f9f9958f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.315+0000 7f9f9958f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.315+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f78005080 con 0x7f9f940a9d70 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f9f8c084f40 con 0x7f9f940a9d70 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f8b7fe640 10 client.4306.objecter ms_dispatch 0x7f9f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f8b7fe640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f8b7fe640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f8b7fe640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f927fc640 10 monclient: get_auth_request con 0x7f9f7406dd60 auth_method 0 2024-03-29T10:50:19.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.316+0000 7f9f8b7fe640 10 client.4306.objecter ms_handle_connect 0x7f9f7406dd60 2024-03-29T10:50:19.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.324+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f8c050820 con 0x7f9f940a9d70 2024-03-29T10:50:19.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.324+0000 7f9f8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:19.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.324+0000 7f9f8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:19.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.636+0000 7f9f9958f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:19.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.636+0000 7f9f9958f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:19.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.636+0000 7f9f9958f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:19.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.636+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f78005330 con 0x7f9f940a9d70 2024-03-29T10:50:19.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.637+0000 7f9f8b7fe640 1 -- 172.21.15.79:0/551573154 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f9f8c059680 con 0x7f9f940a9d70 2024-03-29T10:50:19.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.637+0000 7f9f8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:19.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.637+0000 7f9f8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:19.619 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:19.620 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:19.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.641+0000 7f9f9958f640 10 librados: watch_flush enter 2024-03-29T10:50:19.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.641+0000 7f9f9958f640 10 librados: watch_flush exit 2024-03-29T10:50:19.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.641+0000 7f9f9958f640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:50:19.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.641+0000 7f9f9958f640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 >> v1:172.21.15.79:6816/4290694627 conn(0x7f9f7406dd60 legacy=0x7f9f74070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 10 monclient: shutdown 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 >> v1:172.21.15.82:6789/0 conn(0x7f9f940a9d70 legacy=0x7f9f941db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 shutdown_connections 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 >> 172.21.15.79:0/551573154 conn(0x7f9f94066a00 msgr2=0x7f9f94066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:19.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.642+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 shutdown_connections 2024-03-29T10:50:19.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.643+0000 7f9f9958f640 1 -- 172.21.15.79:0/551573154 wait complete. 2024-03-29T10:50:19.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:19.643+0000 7f9f9958f640 1 librados: shutdown 2024-03-29T10:50:19.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:26.446 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:26.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.612+0000 7f47dbec9640 10 monclient: get_monmap_and_config 2024-03-29T10:50:26.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.612+0000 7f47dbec9640 10 monclient: build_initial_monmap 2024-03-29T10:50:26.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.612+0000 7f47dbec9640 10 monclient: monmap: 2024-03-29T10:50:26.594 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:26.594 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:26.594 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:26.613684+0000 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:26.613684+0000 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:26.595 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.613+0000 7f47dbec9640 1 Processor -- start 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.613+0000 7f47dbec9640 1 -- start start 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.613+0000 7f47dbec9640 10 monclient: init 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.614+0000 7f47dbec9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.614+0000 7f47dbec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.614+0000 7f47dbec9640 10 monclient(hunting): picked mon.noname-c con 0x7f47d41422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:26.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.614+0000 7f47dbec9640 10 monclient(hunting): picked mon.noname-b con 0x7f47d4142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:26.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.615+0000 7f47d9464640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f47d4142690 0x7f47d40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47970/0 (socket says 172.21.15.79:47970) 2024-03-29T10:50:26.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47dbec9640 10 monclient(hunting): picked mon.noname-a con 0x7f47d40b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47d9464640 1 -- 172.21.15.79:0/271748601 learned_addr learned my addr 172.21.15.79:0/271748601 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d4142c80 con 0x7f47d40b17d0 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d40aae40 con 0x7f47d41422b0 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d413fbd0 con 0x7f47d4142690 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47dbec9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.617+0000 7f47dbec9640 10 monclient(hunting): authenticate will time out at 1145.909713s 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225564555 0 0) 0x7f47d413fbd0 con 0x7f47d4142690 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c0003520 con 0x7f47d4142690 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750823589 0 0) 0x7f47d4142c80 con 0x7f47d40b17d0 2024-03-29T10:50:26.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.618+0000 7f47d8c63640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.619+0000 7f47d8c63640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.619+0000 7f47d8c63640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.619+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d413fbd0 con 0x7f47d40b17d0 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.619+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132979384 0 0) 0x7f47d413fbd0 con 0x7f47d40b17d0 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.619+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 >> v1:172.21.15.79:6790/0 conn(0x7f47d4142690 legacy=0x7f47d40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.620+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 >> v1:172.21.15.82:6789/0 conn(0x7f47d41422b0 legacy=0x7f47d413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 10 monclient: _finish_hunting 0 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 1 monclient: found mon.noname-a 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4151940 con 0x7f47d40b17d0 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 10 monclient: _finish_auth 0 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 10 monclient: _check_auth_tickets 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:26.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.621+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f47cc0019f0 con 0x7f47d40b17d0 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.622+0000 7f47dbec9640 5 monclient: authenticate success, global_id 4317 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.622+0000 7f47dbec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.622+0000 7f47d8c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.622+0000 7f47d8c63640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.622+0000 7f47d8c63640 10 monclient: dump: 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:26.605 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:26.606 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:26.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.622+0000 7f47dbec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:26.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47cc001b50 con 0x7f47d40b17d0 2024-03-29T10:50:26.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47d8c63640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47d8c63640 1 -- 172.21.15.79:0/271748601 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f47cc001f20 con 0x7f47d40b17d0 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47d8c63640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47d8c63640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47d8c63640 10 monclient: dump: 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:26.608 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47dbec9640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.624+0000 7f47dbec9640 4 set_mon_vals no callback set 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.625+0000 7f47dbec9640 10 monclient: shutdown 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.625+0000 7f47dbec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.625+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 >> v1:172.21.15.79:6789/0 conn(0x7f47d40b17d0 legacy=0x7f47d40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.625+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 shutdown_connections 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.625+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 >> 172.21.15.79:0/271748601 conn(0x7f47d4066a00 msgr2=0x7f47d4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.626+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 shutdown_connections 2024-03-29T10:50:26.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.626+0000 7f47dbec9640 1 -- 172.21.15.79:0/271748601 wait complete. 2024-03-29T10:50:26.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.627+0000 7f47dbec9640 10 monclient: build_initial_monmap 2024-03-29T10:50:26.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.627+0000 7f47dbec9640 10 monclient: monmap: 2024-03-29T10:50:26.610 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:26.610 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:26.610 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:26.610 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.627+0000 7f47dbec9640 1 librados: starting msgr at 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.627+0000 7f47dbec9640 1 librados: starting objecter 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.627+0000 7f47dbec9640 1 Processor -- start 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.628+0000 7f47dbec9640 1 -- start start 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.628+0000 7f47dbec9640 1 librados: setting wanted keys 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.628+0000 7f47dbec9640 1 librados: calling monclient init 2024-03-29T10:50:26.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.628+0000 7f47dbec9640 10 monclient: init 2024-03-29T10:50:26.612 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.628+0000 7f47dbec9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:26.612 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 10 monclient(hunting): picked mon.noname-b con 0x7f47d40b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 10 monclient(hunting): picked mon.noname-c con 0x7f47d41422b0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 10 monclient(hunting): picked mon.noname-a con 0x7f47d4142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d40acf00 con 0x7f47d4142690 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d40ad1e0 con 0x7f47d40b17d0 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d40abe10 con 0x7f47d41422b0 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:26.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.629+0000 7f47dbec9640 10 monclient(hunting): authenticate will time out at 1145.921920s 2024-03-29T10:50:26.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.630+0000 7f47d9464640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f47d41422b0 0x7f47d412ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:48514/0 (socket says 172.21.15.79:48514) 2024-03-29T10:50:26.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.630+0000 7f47d9464640 1 -- 172.21.15.79:0/3658117889 learned_addr learned my addr 172.21.15.79:0/3658117889 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:26.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.630+0000 7f47ca7fc640 10 client.?.objecter ms_handle_connect 0x7f47d41422b0 2024-03-29T10:50:26.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.630+0000 7f47ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:26.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.632+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928332875 0 0) 0x7f47d40abe10 con 0x7f47d41422b0 2024-03-29T10:50:26.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.632+0000 7f47ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.632+0000 7f47ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b8004eb0 con 0x7f47d41422b0 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 client.?.objecter ms_handle_connect 0x7f47d4142690 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 738905219 0 0) 0x7f47b8004eb0 con 0x7f47d41422b0 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47d40abe10 con 0x7f47d41422b0 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f47d0001970 con 0x7f47d41422b0 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.633+0000 7f47ca7fc640 10 monclient(hunting): dump: 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:26.621 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 client.?.objecter ms_handle_connect 0x7f47d40b17d0 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104122895 0 0) 0x7f47d40acf00 con 0x7f47d4142690 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b8004eb0 con 0x7f47d4142690 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275398262 0 0) 0x7f47d40ad1e0 con 0x7f47d40b17d0 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d40acf00 con 0x7f47d40b17d0 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 849003634 0 0) 0x7f47d40abe10 con 0x7f47d41422b0 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.634+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 >> v1:172.21.15.82:6789/0 conn(0x7f47d40b17d0 legacy=0x7f47d40ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 >> v1:172.21.15.79:6789/0 conn(0x7f47d4142690 legacy=0x7f47d41e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 1 monclient: found mon.c 2024-03-29T10:50:26.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4132c30 con 0x7f47d41422b0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47dbec9640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47dbec9640 10 monclient: _renew_subs 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d0001c90 con 0x7f47d41422b0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47dbec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d40ac2b0 con 0x7f47d41422b0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.635+0000 7f47ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.636+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f47d0004c30 con 0x7f47d41422b0 2024-03-29T10:50:26.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.636+0000 7f47ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.636+0000 7f47dbec9640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.636+0000 7f47ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.636+0000 7f47ca7fc640 10 monclient: dump: 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:26.625 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:26.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.637+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f47d001ac00 con 0x7f47d41422b0 2024-03-29T10:50:26.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.638+0000 7f47dbec9640 10 monclient: _renew_subs 2024-03-29T10:50:26.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.638+0000 7f47dbec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:26.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.638+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d4130850 con 0x7f47d41422b0 2024-03-29T10:50:26.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.638+0000 7f47dbec9640 1 librados: init done 2024-03-29T10:50:26.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.638+0000 7f47cbfff640 4 set_mon_vals no callback set 2024-03-29T10:50:26.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.638+0000 7f47d9c65640 10 monclient: get_auth_request con 0x7f47b806de60 auth_method 0 2024-03-29T10:50:26.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47ca7fc640 10 client.4334.objecter ms_handle_connect 0x7f47b806de60 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47dbec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f47d0054860 con 0x7f47d41422b0 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47ca7fc640 10 client.4334.objecter ms_dispatch 0x7f47d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47ca7fc640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47ca7fc640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.639+0000 7f47ca7fc640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.640+0000 7f47dbec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.640+0000 7f47dbec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:26.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.640+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc005080 con 0x7f47d41422b0 2024-03-29T10:50:26.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.648+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47d0050370 con 0x7f47d41422b0 2024-03-29T10:50:26.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.648+0000 7f47ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:26.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.648+0000 7f47ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.978+0000 7f47dbec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.978+0000 7f47dbec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.978+0000 7f47dbec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:50:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.978+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47bc005330 con 0x7f47d41422b0 2024-03-29T10:50:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.981+0000 7f47ca7fc640 1 -- 172.21.15.79:0/3658117889 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f47d00840e0 con 0x7f47d41422b0 2024-03-29T10:50:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.981+0000 7f47ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.981+0000 7f47ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:26.963 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:26.964 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 10 librados: watch_flush enter 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 10 librados: watch_flush exit 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 10 client.4334.objecter successfully canceled tick 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 >> v1:172.21.15.79:6816/4290694627 conn(0x7f47b806de60 legacy=0x7f47b8070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 10 monclient: shutdown 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 >> v1:172.21.15.79:6790/0 conn(0x7f47d41422b0 legacy=0x7f47d412ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 shutdown_connections 2024-03-29T10:50:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.985+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 >> 172.21.15.79:0/3658117889 conn(0x7f47d4066a00 msgr2=0x7f47d40b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.986+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 shutdown_connections 2024-03-29T10:50:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.986+0000 7f47dbec9640 1 -- 172.21.15.79:0/3658117889 wait complete. 2024-03-29T10:50:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:26.986+0000 7f47dbec9640 1 librados: shutdown 2024-03-29T10:50:26.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:33.794 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:33.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.968+0000 7f0cf7ed9640 10 monclient: get_monmap_and_config 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.968+0000 7f0cf7ed9640 10 monclient: build_initial_monmap 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.968+0000 7f0cf7ed9640 10 monclient: monmap: 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:33.969075+0000 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:33.969075+0000 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:33.950 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:33.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.969+0000 7f0cf7ed9640 1 Processor -- start 2024-03-29T10:50:33.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.969+0000 7f0cf7ed9640 1 -- start start 2024-03-29T10:50:33.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.969+0000 7f0cf7ed9640 10 monclient: init 2024-03-29T10:50:33.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.970+0000 7f0cf7ed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:33.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.970+0000 7f0cf7ed9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:33.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.970+0000 7f0cf7ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf01422b0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:33.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.970+0000 7f0cf7ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf0142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:33.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.972+0000 7f0cf5c75640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f0cf01422b0 0x7f0cf013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:34772/0 (socket says 172.21.15.79:34772) 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.973+0000 7f0cf5474640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f0cf0142690 0x7f0cf00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:36189/0 (socket says 172.21.15.79:36189) 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf7ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf00b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf7ed9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0142c80 con 0x7f0cf0142690 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf7ed9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00aae40 con 0x7f0cf00b17d0 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf7ed9640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf013fbd0 con 0x7f0cf01422b0 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf7ed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf7ed9640 10 monclient(hunting): authenticate will time out at 1153.267921s 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.975+0000 7f0cf5c75640 1 -- 172.21.15.79:0/1899420351 learned_addr learned my addr 172.21.15.79:0/1899420351 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:33.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123653403 0 0) 0x7f0cf0142c80 con 0x7f0cf0142690 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc003520 con 0x7f0cf0142690 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708919664 0 0) 0x7f0cf013fbd0 con 0x7f0cf01422b0 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.976+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0142c80 con 0x7f0cf01422b0 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389481350 0 0) 0x7f0cf0142c80 con 0x7f0cf01422b0 2024-03-29T10:50:33.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 >> v1:172.21.15.82:6789/0 conn(0x7f0cf00b17d0 legacy=0x7f0cf00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 >> v1:172.21.15.79:6789/0 conn(0x7f0cf0142690 legacy=0x7f0cf00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: _finish_hunting 0 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 monclient: found mon.noname-b 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0151940 con 0x7f0cf01422b0 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: _finish_auth 0 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: _check_auth_tickets 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0ce0001520 con 0x7f0cf01422b0 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: dump: 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:33.963 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf7ed9640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf7ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce0001680 con 0x7f0cf01422b0 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf7ed9640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf7ed9640 4 set_mon_vals no callback set 2024-03-29T10:50:33.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 1 -- 172.21.15.79:0/1899420351 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0ce0001a10 con 0x7f0cf01422b0 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.977+0000 7f0cf4c73640 10 monclient: dump: 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.978+0000 7f0cf7ed9640 10 monclient: shutdown 2024-03-29T10:50:33.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.978+0000 7f0cf7ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:33.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.978+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/1899420351 >> v1:172.21.15.79:6790/0 conn(0x7f0cf01422b0 legacy=0x7f0cf013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.979+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/1899420351 shutdown_connections 2024-03-29T10:50:33.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.979+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/1899420351 >> 172.21.15.79:0/1899420351 conn(0x7f0cf0066a00 msgr2=0x7f0cf0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.980+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/1899420351 shutdown_connections 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.981+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/1899420351 wait complete. 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.982+0000 7f0cf7ed9640 10 monclient: build_initial_monmap 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.982+0000 7f0cf7ed9640 10 monclient: monmap: 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:33.967 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.982+0000 7f0cf7ed9640 1 librados: starting msgr at 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.983+0000 7f0cf7ed9640 1 librados: starting objecter 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.983+0000 7f0cf7ed9640 1 Processor -- start 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.983+0000 7f0cf7ed9640 1 -- start start 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.983+0000 7f0cf7ed9640 1 librados: setting wanted keys 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.983+0000 7f0cf7ed9640 1 librados: calling monclient init 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.983+0000 7f0cf7ed9640 10 monclient: init 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.984+0000 7f0cf7ed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:33.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.984+0000 7f0cf7ed9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf00b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf01422b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf0142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf006e970 con 0x7f0cf01422b0 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf006ee00 con 0x7f0cf00b17d0 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0070030 con 0x7f0cf0142690 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf7ed9640 10 monclient(hunting): authenticate will time out at 1153.279854s 2024-03-29T10:50:33.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf5c75640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f0cf00b17d0 0x7f0cf01207c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:44159/0 (socket says 172.21.15.79:44159) 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.987+0000 7f0cf5c75640 1 -- 172.21.15.79:0/73749190 learned_addr learned my addr 172.21.15.79:0/73749190 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 client.?.objecter ms_handle_connect 0x7f0cf00b17d0 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 client.?.objecter ms_handle_connect 0x7f0cf0142690 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686242862 0 0) 0x7f0cf006ee00 con 0x7f0cf00b17d0 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:50:33.972 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd4004eb0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150406411 0 0) 0x7f0cd4004eb0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf006ee00 con 0x7f0cf00b17d0 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0ce00013e0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): dump: 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:33.974 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.989+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190662545 0 0) 0x7f0cf0070030 con 0x7f0cf0142690 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd4004eb0 con 0x7f0cf0142690 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 273035027 0 0) 0x7f0cf006ee00 con 0x7f0cf00b17d0 2024-03-29T10:50:33.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 >> v1:172.21.15.79:6790/0 conn(0x7f0cf0142690 legacy=0x7f0cf0071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.990+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 >> v1:172.21.15.79:6789/0 conn(0x7f0cf01422b0 legacy=0x7f0cf0070a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.991+0000 7f0ce67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 1 monclient: found mon.b 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0070310 con 0x7f0cf00b17d0 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 5 monclient: authenticate success, global_id 4312 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 10 monclient: _renew_subs 2024-03-29T10:50:33.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf0071830 con 0x7f0cf00b17d0 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 10 monclient: _renew_subs 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf01210b0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0cf7ed9640 1 librados: init done 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce0004a90 con 0x7f0cf00b17d0 2024-03-29T10:50:33.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0ce00010e0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce67fc640 10 monclient: dump: 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:33.980 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.992+0000 7f0ce77fe640 4 set_mon_vals no callback set 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.993+0000 7f0cf7ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.993+0000 7f0cf7ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.993+0000 7f0cf7ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.994+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8005080 con 0x7f0cf00b17d0 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.995+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f0ce0004ea0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.996+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f0ce0084bf0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.996+0000 7f0cf5474640 10 monclient: get_auth_request con 0x7f0cd406dd60 auth_method 0 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.996+0000 7f0ce67fc640 10 client.4312.objecter ms_dispatch 0x7f0cf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.996+0000 7f0ce67fc640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.997+0000 7f0ce67fc640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.997+0000 7f0ce67fc640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T10:50:33.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:33.997+0000 7f0ce67fc640 10 client.4312.objecter ms_handle_connect 0x7f0cd406dd60 2024-03-29T10:50:33.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.004+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ce00504d0 con 0x7f0cf00b17d0 2024-03-29T10:50:33.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.004+0000 7f0ce67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:33.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.004+0000 7f0ce67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:34.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.335+0000 7f0cf7ed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:34.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.335+0000 7f0cf7ed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:34.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.335+0000 7f0cf7ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:34.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.335+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cd8005330 con 0x7f0cf00b17d0 2024-03-29T10:50:34.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.336+0000 7f0ce67fc640 1 -- 172.21.15.79:0/73749190 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f0ce0052400 con 0x7f0cf00b17d0 2024-03-29T10:50:34.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.336+0000 7f0ce67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:34.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.336+0000 7f0ce67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:34.318 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:34.318 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.339+0000 7f0cf7ed9640 10 librados: watch_flush enter 2024-03-29T10:50:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 10 librados: watch_flush exit 2024-03-29T10:50:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T10:50:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 10 client.4312.objecter successfully canceled tick 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 >> v1:172.21.15.79:6816/4290694627 conn(0x7f0cd406dd60 legacy=0x7f0cd4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 10 monclient: shutdown 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 >> v1:172.21.15.82:6789/0 conn(0x7f0cf00b17d0 legacy=0x7f0cf01207c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 shutdown_connections 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 >> 172.21.15.79:0/73749190 conn(0x7f0cf0066a00 msgr2=0x7f0cf013da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.340+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 shutdown_connections 2024-03-29T10:50:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.341+0000 7f0cf7ed9640 1 -- 172.21.15.79:0/73749190 wait complete. 2024-03-29T10:50:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:34.341+0000 7f0cf7ed9640 1 librados: shutdown 2024-03-29T10:50:34.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:41.147 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:41.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.311+0000 7f0c47569640 10 monclient: get_monmap_and_config 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.311+0000 7f0c47569640 10 monclient: build_initial_monmap 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.311+0000 7f0c47569640 10 monclient: monmap: 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:41.313051+0000 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:41.313051+0000 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:41.294 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.311+0000 7f0c47569640 1 Processor -- start 2024-03-29T10:50:41.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.312+0000 7f0c47569640 1 -- start start 2024-03-29T10:50:41.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.312+0000 7f0c47569640 10 monclient: init 2024-03-29T10:50:41.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.313+0000 7f0c47569640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:41.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.313+0000 7f0c47569640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:41.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.313+0000 7f0c47569640 10 monclient(hunting): picked mon.noname-a con 0x7f0c400b4820 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:41.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c46567640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f0c400b4820 0x7f0c4013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:35123/0 (socket says 172.21.15.79:35123) 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c45d66640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f0c400a9ce0 0x7f0c40145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:39098/0 (socket says 172.21.15.79:39098) 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c45d66640 1 -- 172.21.15.79:0/4174150937 learned_addr learned my addr 172.21.15.79:0/4174150937 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c47569640 10 monclient(hunting): picked mon.noname-b con 0x7f0c400a9ce0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c47569640 10 monclient(hunting): picked mon.noname-c con 0x7f0c400b3cd0 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c400b4180 con 0x7f0c400b4820 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c400aaf60 con 0x7f0c400b3cd0 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.316+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4013c200 con 0x7f0c400a9ce0 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c47569640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c47569640 10 monclient(hunting): authenticate will time out at 1160.610272s 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532307189 0 0) 0x7f0c4013c200 con 0x7f0c400a9ce0 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient(hunting): _negotiate 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:41.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:41.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:50:41.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c003520 con 0x7f0c400a9ce0 2024-03-29T10:50:41.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133301168 0 0) 0x7f0c2c003520 con 0x7f0c400a9ce0 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 >> v1:172.21.15.82:6789/0 conn(0x7f0c400b3cd0 legacy=0x7f0c4014d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 >> v1:172.21.15.79:6789/0 conn(0x7f0c400b4820 legacy=0x7f0c4013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: _finish_hunting 0 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 monclient: found mon.noname-b 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40145e80 con 0x7f0c400a9ce0 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: _finish_auth 0 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: _check_auth_tickets 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0c30001690 con 0x7f0c400a9ce0 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:41.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.317+0000 7f0c45565640 10 monclient: dump: 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c30001860 con 0x7f0c400a9ce0 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c45565640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c45565640 1 -- 172.21.15.79:0/4174150937 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0c30001cb0 con 0x7f0c400a9ce0 2024-03-29T10:50:41.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c45565640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c45565640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c45565640 10 monclient: dump: 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c47569640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.318+0000 7f0c47569640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:41.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.319+0000 7f0c47569640 4 set_mon_vals no callback set 2024-03-29T10:50:41.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.319+0000 7f0c47569640 10 monclient: shutdown 2024-03-29T10:50:41.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.321+0000 7f0c47569640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:41.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.321+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 >> v1:172.21.15.79:6790/0 conn(0x7f0c400a9ce0 legacy=0x7f0c40145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:41.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.323+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 shutdown_connections 2024-03-29T10:50:41.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.323+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 >> 172.21.15.79:0/4174150937 conn(0x7f0c4006d070 msgr2=0x7f0c40066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:41.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.323+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 shutdown_connections 2024-03-29T10:50:41.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.323+0000 7f0c47569640 1 -- 172.21.15.79:0/4174150937 wait complete. 2024-03-29T10:50:41.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.324+0000 7f0c47569640 10 monclient: build_initial_monmap 2024-03-29T10:50:41.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.324+0000 7f0c47569640 10 monclient: monmap: 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 1 librados: starting msgr at 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 1 librados: starting objecter 2024-03-29T10:50:41.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 1 Processor -- start 2024-03-29T10:50:41.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 1 -- start start 2024-03-29T10:50:41.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 1 librados: setting wanted keys 2024-03-29T10:50:41.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 1 librados: calling monclient init 2024-03-29T10:50:41.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.325+0000 7f0c47569640 10 monclient: init 2024-03-29T10:50:41.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.327+0000 7f0c47569640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:41.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.328+0000 7f0c47569640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:41.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.328+0000 7f0c47569640 10 monclient(hunting): picked mon.noname-c con 0x7f0c400a9ce0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:41.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 10 monclient(hunting): picked mon.noname-a con 0x7f0c400b3cd0 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:41.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 10 monclient(hunting): picked mon.noname-b con 0x7f0c400b4820 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4013b660 con 0x7f0c400b3cd0 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4013b890 con 0x7f0c400b4820 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4013a9a0 con 0x7f0c400a9ce0 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.329+0000 7f0c47569640 10 monclient(hunting): authenticate will time out at 1160.622314s 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.330+0000 7f0c46567640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f0c400a9ce0 0x7f0c401da600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:41590/0 (socket says 172.21.15.79:41590) 2024-03-29T10:50:41.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.330+0000 7f0c46567640 1 -- 172.21.15.79:0/1283641748 learned_addr learned my addr 172.21.15.79:0/1283641748 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:41.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.330+0000 7f0c36ffd640 10 client.?.objecter ms_handle_connect 0x7f0c400a9ce0 2024-03-29T10:50:41.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.330+0000 7f0c36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:41.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.331+0000 7f0c36ffd640 10 client.?.objecter ms_handle_connect 0x7f0c400b3cd0 2024-03-29T10:50:41.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.331+0000 7f0c36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726870867 0 0) 0x7f0c4013b660 con 0x7f0c400b3cd0 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c24004eb0 con 0x7f0c400b3cd0 2024-03-29T10:50:41.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.332+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288337554 0 0) 0x7f0c4013a9a0 con 0x7f0c400a9ce0 2024-03-29T10:50:41.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:41.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:41.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:41.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c4013b660 con 0x7f0c400a9ce0 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 client.?.objecter ms_handle_connect 0x7f0c400b4820 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270668807 0 0) 0x7f0c4013b890 con 0x7f0c400b4820 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:50:41.318 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.334+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c4013a9a0 con 0x7f0c400b4820 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3333388526 0 0) 0x7f0c4013a9a0 con 0x7f0c400b4820 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c4013b890 con 0x7f0c400b4820 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0c38001a90 con 0x7f0c400b4820 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:41.323 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3286378616 0 0) 0x7f0c24004eb0 con 0x7f0c400b3cd0 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c4013a9a0 con 0x7f0c400b3cd0 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0c30001b70 con 0x7f0c400b3cd0 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.335+0000 7f0c36ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:41.324 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1675701877 0 0) 0x7f0c4013b660 con 0x7f0c400a9ce0 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c24004eb0 con 0x7f0c400a9ce0 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0c3c001850 con 0x7f0c400a9ce0 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.325 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3036431633 0 0) 0x7f0c4013b890 con 0x7f0c400b4820 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 >> v1:172.21.15.79:6790/0 conn(0x7f0c400a9ce0 legacy=0x7f0c401da600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 >> v1:172.21.15.79:6789/0 conn(0x7f0c400b3cd0 legacy=0x7f0c40133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 monclient: found mon.b 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c401dad10 con 0x7f0c400b4820 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:41.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 10 monclient: _renew_subs 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c40135d00 con 0x7f0c400b4820 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 10 monclient: _renew_subs 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c401362a0 con 0x7f0c400b4820 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.336+0000 7f0c47569640 1 librados: init done 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.337+0000 7f0c47569640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c47569640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c47569640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4004f730 con 0x7f0c400b4820 2024-03-29T10:50:41.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c38001bf0 con 0x7f0c400b4820 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0c38003a30 con 0x7f0c400b4820 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c36ffd640 10 monclient: dump: 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:41.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.338+0000 7f0c44d64640 4 set_mon_vals no callback set 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.339+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f0c38019a80 con 0x7f0c400b4820 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.339+0000 7f0c46567640 10 monclient: get_auth_request con 0x7f0c2406e080 auth_method 0 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.339+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f0c380839d0 con 0x7f0c400b4820 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.339+0000 7f0c36ffd640 10 client.4315.objecter ms_dispatch 0x7f0c40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.339+0000 7f0c36ffd640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.339+0000 7f0c36ffd640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.340+0000 7f0c36ffd640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T10:50:41.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.340+0000 7f0c36ffd640 10 client.4315.objecter ms_handle_connect 0x7f0c2406e080 2024-03-29T10:50:41.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.345+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c3804f0e0 con 0x7f0c400b4820 2024-03-29T10:50:41.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.345+0000 7f0c36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:41.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.345+0000 7f0c36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:41.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.663+0000 7f0c47569640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:41.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.663+0000 7f0c47569640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:41.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.663+0000 7f0c47569640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:41.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.663+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c40074d80 con 0x7f0c400b4820 2024-03-29T10:50:41.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.663+0000 7f0c36ffd640 1 -- 172.21.15.79:0/1283641748 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f0c380511e0 con 0x7f0c400b4820 2024-03-29T10:50:41.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.664+0000 7f0c36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:41.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.664+0000 7f0c36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:41.647 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:41.647 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:41.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.667+0000 7f0c47569640 10 librados: watch_flush enter 2024-03-29T10:50:41.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.667+0000 7f0c47569640 10 librados: watch_flush exit 2024-03-29T10:50:41.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.667+0000 7f0c47569640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T10:50:41.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.667+0000 7f0c47569640 10 client.4315.objecter successfully canceled tick 2024-03-29T10:50:41.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.667+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 >> v1:172.21.15.79:6816/4290694627 conn(0x7f0c2406e080 legacy=0x7f0c24070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 10 monclient: shutdown 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 >> v1:172.21.15.82:6789/0 conn(0x7f0c400b4820 legacy=0x7f0c401e18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 shutdown_connections 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 >> 172.21.15.79:0/1283641748 conn(0x7f0c4006d070 msgr2=0x7f0c400b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 shutdown_connections 2024-03-29T10:50:41.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 1 -- 172.21.15.79:0/1283641748 wait complete. 2024-03-29T10:50:41.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:41.668+0000 7f0c47569640 1 librados: shutdown 2024-03-29T10:50:41.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:48.476 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:48.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.643+0000 7f4260fcd640 10 monclient: get_monmap_and_config 2024-03-29T10:50:48.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.643+0000 7f4260fcd640 10 monclient: build_initial_monmap 2024-03-29T10:50:48.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.643+0000 7f4260fcd640 10 monclient: monmap: 2024-03-29T10:50:48.625 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:48.625 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:48.644105+0000 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:48.644105+0000 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:48.626 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:48.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.648+0000 7f4260fcd640 1 Processor -- start 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.648+0000 7f4260fcd640 1 -- start start 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.648+0000 7f4260fcd640 10 monclient: init 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient(hunting): picked mon.noname-a con 0x7f425c0b1f60 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient(hunting): picked mon.noname-b con 0x7f425c071eb0 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient(hunting): picked mon.noname-c con 0x7f425c143d10 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c06ef80 con 0x7f425c0b1f60 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c06f260 con 0x7f425c143d10 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c0b40c0 con 0x7f425c071eb0 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:48.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.649+0000 7f4260fcd640 10 monclient(hunting): authenticate will time out at 1167.941537s 2024-03-29T10:50:48.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.652+0000 7f4259d9b640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f425c071eb0 0x7f425c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35981/0 (socket says 172.21.15.79:35981) 2024-03-29T10:50:48.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.652+0000 7f4259d9b640 1 -- 172.21.15.79:0/1613554434 learned_addr learned my addr 172.21.15.79:0/1613554434 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:48.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.656+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817920544 0 0) 0x7f425c0b40c0 con 0x7f425c071eb0 2024-03-29T10:50:48.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.656+0000 7f425959a640 10 monclient(hunting): _negotiate 2024-03-29T10:50:48.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.656+0000 7f425959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:48.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.656+0000 7f425959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:48.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.656+0000 7f425959a640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:50:48.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.656+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4244003520 con 0x7f425c071eb0 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3247580400 0 0) 0x7f4244003520 con 0x7f425c071eb0 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 >> v1:172.21.15.82:6789/0 conn(0x7f425c143d10 legacy=0x7f425c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 >> v1:172.21.15.79:6789/0 conn(0x7f425c0b1f60 legacy=0x7f425c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: _finish_hunting 0 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 monclient: found mon.noname-b 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c155d70 con 0x7f425c071eb0 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: _finish_auth 0 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: _check_auth_tickets 2024-03-29T10:50:48.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4250001690 con 0x7f425c071eb0 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: dump: 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:48.641 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:48.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42500017f0 con 0x7f425c071eb0 2024-03-29T10:50:48.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:48.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 1 -- 172.21.15.79:0/1613554434 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4250001c00 con 0x7f425c071eb0 2024-03-29T10:50:48.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:48.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f425959a640 10 monclient: dump: 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:48.643 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:48.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f4260fcd640 5 monclient: authenticate success, global_id 4349 2024-03-29T10:50:48.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f4260fcd640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:48.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.658+0000 7f4260fcd640 4 set_mon_vals no callback set 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.659+0000 7f4260fcd640 10 monclient: shutdown 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.659+0000 7f4260fcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.659+0000 7f4260fcd640 1 -- 172.21.15.79:0/1613554434 >> v1:172.21.15.79:6790/0 conn(0x7f425c071eb0 legacy=0x7f425c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.662+0000 7f4260fcd640 1 -- 172.21.15.79:0/1613554434 shutdown_connections 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.662+0000 7f4260fcd640 1 -- 172.21.15.79:0/1613554434 >> 172.21.15.79:0/1613554434 conn(0x7f425c066a00 msgr2=0x7f425c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.663+0000 7f4260fcd640 1 -- 172.21.15.79:0/1613554434 shutdown_connections 2024-03-29T10:50:48.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.663+0000 7f4260fcd640 1 -- 172.21.15.79:0/1613554434 wait complete. 2024-03-29T10:50:48.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.664+0000 7f4260fcd640 10 monclient: build_initial_monmap 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.664+0000 7f4260fcd640 10 monclient: monmap: 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.664+0000 7f4260fcd640 1 librados: starting msgr at 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.664+0000 7f4260fcd640 1 librados: starting objecter 2024-03-29T10:50:48.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.664+0000 7f4260fcd640 1 Processor -- start 2024-03-29T10:50:48.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.665+0000 7f4260fcd640 1 -- start start 2024-03-29T10:50:48.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.665+0000 7f4260fcd640 1 librados: setting wanted keys 2024-03-29T10:50:48.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.665+0000 7f4260fcd640 1 librados: calling monclient init 2024-03-29T10:50:48.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.665+0000 7f4260fcd640 10 monclient: init 2024-03-29T10:50:48.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.666+0000 7f4260fcd640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:48.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.666+0000 7f4260fcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:48.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.668+0000 7f425a59c640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f425c0b1f60 0x7f425c1402c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:36073/0 (socket says 172.21.15.79:36073) 2024-03-29T10:50:48.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.668+0000 7f425a59c640 1 -- 172.21.15.79:0/962421748 learned_addr learned my addr 172.21.15.79:0/962421748 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:48.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.668+0000 7f4260fcd640 10 monclient(hunting): picked mon.noname-c con 0x7f425c0b1f60 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:48.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.668+0000 7f4260fcd640 10 monclient(hunting): picked mon.noname-a con 0x7f425c143d10 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:48.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f4260fcd640 10 monclient(hunting): picked mon.noname-b con 0x7f425c152050 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:48.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f4260fcd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c13fe30 con 0x7f425c143d10 2024-03-29T10:50:48.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f4260fcd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c1409d0 con 0x7f425c152050 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f4260fcd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c13f810 con 0x7f425c0b1f60 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f4260fcd640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f4260fcd640 10 monclient(hunting): authenticate will time out at 1167.964185s 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.669+0000 7f424affd640 10 client.?.objecter ms_handle_connect 0x7f425c0b1f60 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 client.?.objecter ms_handle_connect 0x7f425c143d10 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801100595 0 0) 0x7f425c13fe30 con 0x7f425c143d10 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:50:48.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.672+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423c004eb0 con 0x7f425c143d10 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546583644 0 0) 0x7f425c13f810 con 0x7f425c0b1f60 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c13fe30 con 0x7f425c0b1f60 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732880568 0 0) 0x7f423c004eb0 con 0x7f425c143d10 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f425c13f810 con 0x7f425c143d10 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4250001b00 con 0x7f425c143d10 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:48.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): dump: 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015678380 0 0) 0x7f425c13fe30 con 0x7f425c0b1f60 2024-03-29T10:50:48.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f423c004eb0 con 0x7f425c0b1f60 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2252221111 0 0) 0x7f425c13f810 con 0x7f425c143d10 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 >> v1:172.21.15.79:6790/0 conn(0x7f425c0b1f60 legacy=0x7f425c1402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.674+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 >> v1:172.21.15.82:6789/0 conn(0x7f425c152050 legacy=0x7f425c13f100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 1 monclient: found mon.a 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c13fb00 con 0x7f425c143d10 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 10 monclient: _finish_auth 0 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.675+0000 7f424affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:50:48.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 10 monclient: _renew_subs 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c14b450 con 0x7f425c143d10 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 10 monclient: _renew_subs 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c070fe0 con 0x7f425c143d10 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.676+0000 7f4260fcd640 1 librados: init done 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.677+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4250004a90 con 0x7f425c143d10 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.677+0000 7f424affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.677+0000 7f424bfff640 4 set_mon_vals no callback set 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4250004da0 con 0x7f425c143d10 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f424affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:48.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f424affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f424affd640 10 monclient: dump: 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f4248ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f4248ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:48.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f4248ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:50:48.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.678+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c075f10 con 0x7f425c143d10 2024-03-29T10:50:48.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.679+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f425001ad70 con 0x7f425c143d10 2024-03-29T10:50:48.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.679+0000 7f425a59c640 10 monclient: get_auth_request con 0x7f423c06ddc0 auth_method 0 2024-03-29T10:50:48.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.680+0000 7f424affd640 10 client.4332.objecter ms_handle_connect 0x7f423c06ddc0 2024-03-29T10:50:48.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.680+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f4250084a30 con 0x7f425c143d10 2024-03-29T10:50:48.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.680+0000 7f424affd640 10 client.4332.objecter ms_dispatch 0x7f425c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:48.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.680+0000 7f424affd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:48.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.680+0000 7f424affd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:48.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.680+0000 7f424affd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T10:50:48.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.687+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4250050310 con 0x7f425c143d10 2024-03-29T10:50:48.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.687+0000 7f424affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:48.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:48.687+0000 7f424affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:48.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.009+0000 7f4248ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:48.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.009+0000 7f4248ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:48.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.009+0000 7f4248ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:50:48.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.009+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f425c0b8a60 con 0x7f425c143d10 2024-03-29T10:50:48.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.010+0000 7f424affd640 1 -- 172.21.15.79:0/962421748 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f4250052240 con 0x7f425c143d10 2024-03-29T10:50:48.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.010+0000 7f424affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:48.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.010+0000 7f424affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:48.996 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:48.996 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:49.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 10 librados: watch_flush enter 2024-03-29T10:50:49.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 10 librados: watch_flush exit 2024-03-29T10:50:49.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T10:50:49.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 10 client.4332.objecter successfully canceled tick 2024-03-29T10:50:49.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 >> v1:172.21.15.79:6816/4290694627 conn(0x7f423c06ddc0 legacy=0x7f423c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:49.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 10 monclient: shutdown 2024-03-29T10:50:49.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:49.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.017+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 >> v1:172.21.15.79:6789/0 conn(0x7f425c143d10 legacy=0x7f425c13e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:49.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.018+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 shutdown_connections 2024-03-29T10:50:49.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.018+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 >> 172.21.15.79:0/962421748 conn(0x7f425c066a00 msgr2=0x7f425c142860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:49.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.018+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 shutdown_connections 2024-03-29T10:50:49.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.018+0000 7f4248ff9640 1 -- 172.21.15.79:0/962421748 wait complete. 2024-03-29T10:50:49.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:49.018+0000 7f4248ff9640 1 librados: shutdown 2024-03-29T10:50:49.019 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:55.826 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:55.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.994+0000 7fda0ea9a640 10 monclient: get_monmap_and_config 2024-03-29T10:50:55.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.994+0000 7fda0ea9a640 10 monclient: build_initial_monmap 2024-03-29T10:50:55.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.994+0000 7fda0ea9a640 10 monclient: monmap: 2024-03-29T10:50:55.977 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:55.977 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:50:55.995700+0000 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:50:55.995700+0000 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:50:55.978 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:55.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.996+0000 7fda0ea9a640 1 Processor -- start 2024-03-29T10:50:55.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.996+0000 7fda0ea9a640 1 -- start start 2024-03-29T10:50:55.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.996+0000 7fda0ea9a640 10 monclient: init 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient(hunting): picked mon.noname-a con 0x7fda080b4100 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient(hunting): picked mon.noname-b con 0x7fda0813ee90 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient(hunting): picked mon.noname-c con 0x7fda080af900 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda080afd10 con 0x7fda080b4100 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda080af6f0 con 0x7fda080af900 2024-03-29T10:50:55.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda080aac80 con 0x7fda0813ee90 2024-03-29T10:50:55.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:55.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:55.997+0000 7fda0ea9a640 10 monclient(hunting): authenticate will time out at 1175.289732s 2024-03-29T10:50:55.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.003+0000 7fda07fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fda0813ee90 0x7fda0813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47717/0 (socket says 172.21.15.79:47717) 2024-03-29T10:50:55.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.003+0000 7fda07fff640 1 -- 172.21.15.79:0/3418169408 learned_addr learned my addr 172.21.15.79:0/3418169408 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:55.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630624515 0 0) 0x7fda080aac80 con 0x7fda0813ee90 2024-03-29T10:50:55.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:55.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:55.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:55.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:50:55.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9f4003520 con 0x7fda0813ee90 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750730733 0 0) 0x7fda080afd10 con 0x7fda080b4100 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.004+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda080aac80 con 0x7fda080b4100 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.005+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611947538 0 0) 0x7fda080aac80 con 0x7fda080b4100 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.005+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 >> v1:172.21.15.79:6790/0 conn(0x7fda0813ee90 legacy=0x7fda0813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:55.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.005+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 >> v1:172.21.15.82:6789/0 conn(0x7fda080af900 legacy=0x7fda0814be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 1 monclient: found mon.noname-a 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda080aaf60 con 0x7fda080b4100 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd9f8001970 con 0x7fda080b4100 2024-03-29T10:50:55.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: dump: 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda0ea9a640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda0ea9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:55.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f8001ad0 con 0x7fda080b4100 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 1 -- 172.21.15.79:0/3418169408 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd9f8001f20 con 0x7fda080b4100 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.008+0000 7fda077fe640 10 monclient: dump: 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:55.992 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.009+0000 7fda0ea9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.009+0000 7fda0ea9a640 4 set_mon_vals no callback set 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.010+0000 7fda0ea9a640 10 monclient: shutdown 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.010+0000 7fda0ea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.010+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3418169408 >> v1:172.21.15.79:6789/0 conn(0x7fda080b4100 legacy=0x7fda080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.010+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3418169408 shutdown_connections 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.010+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3418169408 >> 172.21.15.79:0/3418169408 conn(0x7fda080b35e0 msgr2=0x7fda080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:55.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.011+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3418169408 shutdown_connections 2024-03-29T10:50:55.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.012+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3418169408 wait complete. 2024-03-29T10:50:55.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.012+0000 7fda0ea9a640 10 monclient: build_initial_monmap 2024-03-29T10:50:55.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.012+0000 7fda0ea9a640 10 monclient: monmap: 2024-03-29T10:50:55.995 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:50:55.995 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 1 librados: starting msgr at 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 1 librados: starting objecter 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 1 Processor -- start 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 1 -- start start 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 1 librados: setting wanted keys 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 1 librados: calling monclient init 2024-03-29T10:50:55.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.013+0000 7fda0ea9a640 10 monclient: init 2024-03-29T10:50:55.997 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.014+0000 7fda0ea9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:55.997 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.014+0000 7fda0ea9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:56.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.015+0000 7fda0ea9a640 10 monclient(hunting): picked mon.noname-b con 0x7fda080af900 addr v1:172.21.15.82:6789/0 2024-03-29T10:50:56.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 10 monclient(hunting): picked mon.noname-a con 0x7fda080b4100 addr v1:172.21.15.79:6789/0 2024-03-29T10:50:56.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 10 monclient(hunting): picked mon.noname-c con 0x7fda0813ee90 addr v1:172.21.15.79:6790/0 2024-03-29T10:50:56.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda080b3390 con 0x7fda080b4100 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda08128da0 con 0x7fda080af900 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda08129230 con 0x7fda0813ee90 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0ea9a640 10 monclient(hunting): authenticate will time out at 1175.313551s 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0c836640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fda080af900 0x7fda080b11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:38969/0 (socket says 172.21.15.79:38969) 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.021+0000 7fda0c836640 1 -- 172.21.15.79:0/3301514399 learned_addr learned my addr 172.21.15.79:0/3301514399 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 client.?.objecter ms_handle_connect 0x7fda080af900 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444499187 0 0) 0x7fda08128da0 con 0x7fda080af900 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e8004eb0 con 0x7fda080af900 2024-03-29T10:50:56.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 client.?.objecter ms_handle_connect 0x7fda080b4100 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066321935 0 0) 0x7fd9e8004eb0 con 0x7fda080af900 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9e8005300 con 0x7fda080af900 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd9f8001f20 con 0x7fda080af900 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): dump: 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:56.007 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.022+0000 7fda057fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.023+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494677918 0 0) 0x7fda080b3390 con 0x7fda080b4100 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.023+0000 7fda057fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.023+0000 7fda057fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.023+0000 7fda057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.023+0000 7fda057fa640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:50:56.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.023+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e8005850 con 0x7fda080b4100 2024-03-29T10:50:56.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7479767 0 0) 0x7fd9e8005850 con 0x7fda080b4100 2024-03-29T10:50:56.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda080b3390 con 0x7fda080b4100 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd9f0001850 con 0x7fda080b4100 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 10 monclient(hunting): dump: 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:56.010 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 10 client.?.objecter ms_handle_connect 0x7fda0813ee90 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.025+0000 7fda057fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2898123441 0 0) 0x7fd9e8005300 con 0x7fda080af900 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 >> v1:172.21.15.79:6790/0 conn(0x7fda0813ee90 legacy=0x7fda081e7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 >> v1:172.21.15.79:6789/0 conn(0x7fda080b4100 legacy=0x7fda08127900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 1 monclient: found mon.b 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda080b2250 con 0x7fda080af900 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:56.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 5 monclient: authenticate success, global_id 4321 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 10 monclient: _renew_subs 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda080b18c0 con 0x7fda080af900 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 10 monclient: _renew_subs 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda081281f0 con 0x7fda080af900 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.026+0000 7fda0ea9a640 1 librados: init done 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9f8003c20 con 0x7fda080af900 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd9f8001f20 con 0x7fda080af900 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:56.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda057fa640 10 monclient: dump: 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:50:56.013 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fd9feffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.028+0000 7fda067fc640 4 set_mon_vals no callback set 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.029+0000 7fd9feffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.029+0000 7fd9feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.029+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda080b24a0 con 0x7fda080af900 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fd9f8003e10 con 0x7fda080af900 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fd9f8084c60 con 0x7fda080af900 2024-03-29T10:50:56.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda057fa640 10 client.4321.objecter ms_dispatch 0x7fda08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:56.015 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda057fa640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:56.015 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda057fa640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:56.015 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda057fa640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T10:50:56.015 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.030+0000 7fda07fff640 10 monclient: get_auth_request con 0x7fd9e806e450 auth_method 0 2024-03-29T10:50:56.015 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.031+0000 7fda057fa640 10 client.4321.objecter ms_handle_connect 0x7fd9e806e450 2024-03-29T10:50:56.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.036+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9f8050540 con 0x7fda080af900 2024-03-29T10:50:56.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.036+0000 7fda057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:56.018 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.036+0000 7fda057fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:56.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.366+0000 7fd9feffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:56.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.366+0000 7fd9feffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:56.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.366+0000 7fd9feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:50:56.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.366+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda0806dfd0 con 0x7fda080af900 2024-03-29T10:50:56.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.367+0000 7fda057fa640 1 -- 172.21.15.79:0/3301514399 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fd9f8050540 con 0x7fda080af900 2024-03-29T10:50:56.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.367+0000 7fda057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:56.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.367+0000 7fda057fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:56.349 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:50:56.349 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:56.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 10 librados: watch_flush enter 2024-03-29T10:50:56.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 10 librados: watch_flush exit 2024-03-29T10:50:56.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T10:50:56.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 10 client.4321.objecter successfully canceled tick 2024-03-29T10:50:56.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 >> v1:172.21.15.79:6816/4290694627 conn(0x7fd9e806e450 legacy=0x7fd9e8070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:56.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 10 monclient: shutdown 2024-03-29T10:50:56.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:56.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 >> v1:172.21.15.82:6789/0 conn(0x7fda080af900 legacy=0x7fda080b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:56.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 shutdown_connections 2024-03-29T10:50:56.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.370+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 >> 172.21.15.79:0/3301514399 conn(0x7fda080b35e0 msgr2=0x7fda080acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:56.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.375+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 shutdown_connections 2024-03-29T10:50:56.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.375+0000 7fd9feffd640 1 -- 172.21.15.79:0/3301514399 wait complete. 2024-03-29T10:50:56.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:50:56.375+0000 7fd9feffd640 1 librados: shutdown 2024-03-29T10:50:56.375 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:03.184 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:03.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.345+0000 7f8fb6fab640 10 monclient: get_monmap_and_config 2024-03-29T10:51:03.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.345+0000 7f8fb6fab640 10 monclient: build_initial_monmap 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.345+0000 7f8fb6fab640 10 monclient: monmap: 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:03.345967+0000 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:03.345967+0000 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:03.327 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:03.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.346+0000 7f8fb6fab640 1 Processor -- start 2024-03-29T10:51:03.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.346+0000 7f8fb6fab640 1 -- start start 2024-03-29T10:51:03.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.346+0000 7f8fb6fab640 10 monclient: init 2024-03-29T10:51:03.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa8036a00 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa802d2f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa802c7d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa802e330 con 0x7f8fa8036a00 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa802cdd0 con 0x7f8fa802d2f0 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80c5d80 con 0x7f8fa802c7d0 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb6fab640 10 monclient(hunting): authenticate will time out at 1182.639571s 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb5fa9640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f8fa8036a00 0x7f8fa80c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:33968/0 (socket says 172.21.15.79:33968) 2024-03-29T10:51:03.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.347+0000 7f8fb5fa9640 1 -- 172.21.15.79:0/221268727 learned_addr learned my addr 172.21.15.79:0/221268727 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213498416 0 0) 0x7f8fa802e330 con 0x7f8fa8036a00 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): _negotiate 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c003520 con 0x7f8fa8036a00 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958847535 0 0) 0x7f8fa80c5d80 con 0x7f8fa802c7d0 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): _negotiate 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:51:03.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa802e330 con 0x7f8fa802c7d0 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067716028 0 0) 0x7f8f9c003520 con 0x7f8fa8036a00 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 >> v1:172.21.15.79:6790/0 conn(0x7f8fa802c7d0 legacy=0x7f8fa80d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 >> v1:172.21.15.82:6789/0 conn(0x7f8fa802d2f0 legacy=0x7f8fa80c2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: _finish_hunting 0 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 monclient: found mon.noname-a 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80c5f10 con 0x7f8fa8036a00 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: _finish_auth 0 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: _check_auth_tickets 2024-03-29T10:51:03.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8fa0001690 con 0x7f8fa8036a00 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.348+0000 7f8fb4fa7640 10 monclient: dump: 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:03.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb6fab640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb6fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fa0001920 con 0x7f8fa8036a00 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb4fa7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb4fa7640 1 -- 172.21.15.79:0/221268727 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8fa0001cf0 con 0x7f8fa8036a00 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb4fa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb4fa7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb4fa7640 10 monclient: dump: 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:03.334 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb6fab640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.349+0000 7f8fb6fab640 4 set_mon_vals no callback set 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 10 monclient: shutdown 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 1 -- 172.21.15.79:0/221268727 >> v1:172.21.15.79:6789/0 conn(0x7f8fa8036a00 legacy=0x7f8fa80c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 1 -- 172.21.15.79:0/221268727 shutdown_connections 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 1 -- 172.21.15.79:0/221268727 >> 172.21.15.79:0/221268727 conn(0x7f8fa8005c60 msgr2=0x7f8fa8006070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 1 -- 172.21.15.79:0/221268727 shutdown_connections 2024-03-29T10:51:03.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.350+0000 7f8fb6fab640 1 -- 172.21.15.79:0/221268727 wait complete. 2024-03-29T10:51:03.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.351+0000 7f8fb6fab640 10 monclient: build_initial_monmap 2024-03-29T10:51:03.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.351+0000 7f8fb6fab640 10 monclient: monmap: 2024-03-29T10:51:03.341 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:03.341 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.351+0000 7f8fb6fab640 1 librados: starting msgr at 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.351+0000 7f8fb6fab640 1 librados: starting objecter 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.351+0000 7f8fb6fab640 1 Processor -- start 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 1 -- start start 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 1 librados: setting wanted keys 2024-03-29T10:51:03.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 1 librados: calling monclient init 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient: init 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa802c7d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa802d2f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa8036a00 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80c9570 con 0x7f8fa8036a00 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80c9850 con 0x7f8fa802d2f0 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80c8d70 con 0x7f8fa802c7d0 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.352+0000 7f8fb6fab640 10 monclient(hunting): authenticate will time out at 1182.645050s 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fb67aa640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f8fa8036a00 0x7f8fa816c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:35754/0 (socket says 172.21.15.79:35754) 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fb57a8640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f8fa802d2f0 0x7f8fa80bb810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:53322/0 (socket says 172.21.15.79:53322) 2024-03-29T10:51:03.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fb57a8640 1 -- 172.21.15.79:0/2174693547 learned_addr learned my addr 172.21.15.79:0/2174693547 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 client.?.objecter ms_handle_connect 0x7f8fa8036a00 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 client.?.objecter ms_handle_connect 0x7f8fa802c7d0 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 client.?.objecter ms_handle_connect 0x7f8fa802d2f0 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366202819 0 0) 0x7f8fa80c9570 con 0x7f8fa8036a00 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:03.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.353+0000 7f8fa67fc640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94004eb0 con 0x7f8fa8036a00 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194710919 0 0) 0x7f8fa80c9850 con 0x7f8fa802d2f0 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa80c9570 con 0x7f8fa802d2f0 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1972713803 0 0) 0x7f8fa80c9570 con 0x7f8fa802d2f0 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa80c9850 con 0x7f8fa802d2f0 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8fac001a30 con 0x7f8fa802d2f0 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:03.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): dump: 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:03.346 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679872030 0 0) 0x7f8fa80c8d70 con 0x7f8fa802c7d0 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa80c9570 con 0x7f8fa802c7d0 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3160514276 0 0) 0x7f8f94004eb0 con 0x7f8fa8036a00 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa80c8d70 con 0x7f8fa8036a00 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2318344628 0 0) 0x7f8fa80c9850 con 0x7f8fa802d2f0 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 >> v1:172.21.15.79:6790/0 conn(0x7f8fa802c7d0 legacy=0x7f8fa80be500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 >> v1:172.21.15.79:6789/0 conn(0x7f8fa8036a00 legacy=0x7f8fa816c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:03.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 monclient: found mon.b 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.354+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80bc3f0 con 0x7f8fa802d2f0 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 10 monclient: _renew_subs 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa80bec10 con 0x7f8fa802d2f0 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 10 monclient: _renew_subs 2024-03-29T10:51:03.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa80bc100 con 0x7f8fa802d2f0 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fb6fab640 1 librados: init done 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fac001b90 con 0x7f8fa802d2f0 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8fac003a60 con 0x7f8fa802d2f0 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa67fc640 10 monclient: dump: 2024-03-29T10:51:03.349 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.355+0000 7f8fa7fff640 4 set_mon_vals no callback set 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.357+0000 7f8fb6fab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.357+0000 7f8fb6fab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.357+0000 7f8fb6fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:03.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.357+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f98005080 con 0x7f8fa802d2f0 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.358+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f8fac003bc0 con 0x7f8fa802d2f0 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.358+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f8fac084ac0 con 0x7f8fa802d2f0 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.359+0000 7f8fa67fc640 10 client.4327.objecter ms_dispatch 0x7f8fb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.359+0000 7f8fa67fc640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.359+0000 7f8fa67fc640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.359+0000 7f8fa67fc640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.359+0000 7f8fb5fa9640 10 monclient: get_auth_request con 0x7f8f9406dd80 auth_method 0 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.360+0000 7f8fa67fc640 10 client.4327.objecter ms_handle_connect 0x7f8f9406dd80 2024-03-29T10:51:03.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.364+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fac0503a0 con 0x7f8fa802d2f0 2024-03-29T10:51:03.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.364+0000 7f8fa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:03.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.364+0000 7f8fa67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:03.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.680+0000 7f8fb6fab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:03.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.680+0000 7f8fb6fab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:03.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.680+0000 7f8fb6fab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:03.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.680+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f98005330 con 0x7f8fa802d2f0 2024-03-29T10:51:03.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.681+0000 7f8fa67fc640 1 -- 172.21.15.79:0/2174693547 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f8fac0522d0 con 0x7f8fa802d2f0 2024-03-29T10:51:03.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.681+0000 7f8fa67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:03.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.681+0000 7f8fa67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:03.663 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:03.663 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:03.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.684+0000 7f8fb6fab640 10 librados: watch_flush enter 2024-03-29T10:51:03.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 10 librados: watch_flush exit 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 10 client.4327.objecter successfully canceled tick 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 >> v1:172.21.15.79:6816/4290694627 conn(0x7f8f9406dd80 legacy=0x7f8f94070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 10 monclient: shutdown 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 >> v1:172.21.15.82:6789/0 conn(0x7f8fa802d2f0 legacy=0x7f8fa80bb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 shutdown_connections 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 >> 172.21.15.79:0/2174693547 conn(0x7f8fa8005c60 msgr2=0x7f8fa803a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 shutdown_connections 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 -- 172.21.15.79:0/2174693547 wait complete. 2024-03-29T10:51:03.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:03.685+0000 7f8fb6fab640 1 librados: shutdown 2024-03-29T10:51:03.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:10.489 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.673+0000 7f0288f86640 10 monclient: get_monmap_and_config 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.673+0000 7f0288f86640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.673+0000 7f0288f86640 10 monclient: monmap: 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:10.674376+0000 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:10.674376+0000 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.655 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:10.656 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:10.656 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:10.656 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:10.656 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.674+0000 7f0288f86640 1 Processor -- start 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.674+0000 7f0288f86640 1 -- start start 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.674+0000 7f0288f86640 10 monclient: init 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient(hunting): picked mon.noname-a con 0x7f02840b2800 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:10.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient(hunting): picked mon.noname-c con 0x7f0284142b20 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient(hunting): picked mon.noname-b con 0x7f028413c040 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02840aad10 con 0x7f02840b2800 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0284141f30 con 0x7f0284142b20 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02840aaf60 con 0x7f028413c040 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0288f86640 10 monclient(hunting): authenticate will time out at 1189.967793s 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0283fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f028413c040 0x7f028414d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47172/0 (socket says 172.21.15.79:47172) 2024-03-29T10:51:10.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.675+0000 7f0283fff640 1 -- 172.21.15.79:0/501028713 learned_addr learned my addr 172.21.15.79:0/501028713 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:10.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986144615 0 0) 0x7f0284141f30 con 0x7f0284142b20 2024-03-29T10:51:10.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0270003520 con 0x7f0284142b20 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480803766 0 0) 0x7f02840aad10 con 0x7f02840b2800 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.676+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0284141f30 con 0x7f02840b2800 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197749922 0 0) 0x7f0270003520 con 0x7f0284142b20 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 >> v1:172.21.15.79:6790/0 conn(0x7f028413c040 legacy=0x7f028414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 >> v1:172.21.15.79:6789/0 conn(0x7f02840b2800 legacy=0x7f0284142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 monclient: found mon.noname-c 2024-03-29T10:51:10.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840b3900 con 0x7f0284142b20 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f026c001690 con 0x7f0284142b20 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: dump: 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.662 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0288f86640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0288f86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f026c0017f0 con 0x7f0284142b20 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0282ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0288f86640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.677+0000 7f0288f86640 4 set_mon_vals no callback set 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0282ffd640 1 -- 172.21.15.79:0/501028713 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f026c001c40 con 0x7f0284142b20 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0282ffd640 10 monclient: dump: 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0288f86640 10 monclient: shutdown 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0288f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0288f86640 1 -- 172.21.15.79:0/501028713 >> v1:172.21.15.82:6789/0 conn(0x7f0284142b20 legacy=0x7f028413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0288f86640 1 -- 172.21.15.79:0/501028713 shutdown_connections 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.678+0000 7f0288f86640 1 -- 172.21.15.79:0/501028713 >> 172.21.15.79:0/501028713 conn(0x7f028406d0e0 msgr2=0x7f0284066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:10.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.679+0000 7f0288f86640 1 -- 172.21.15.79:0/501028713 shutdown_connections 2024-03-29T10:51:10.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.679+0000 7f0288f86640 1 -- 172.21.15.79:0/501028713 wait complete. 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.680+0000 7f0288f86640 10 monclient: build_initial_monmap 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.680+0000 7f0288f86640 10 monclient: monmap: 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.680+0000 7f0288f86640 1 librados: starting msgr at 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.680+0000 7f0288f86640 1 librados: starting objecter 2024-03-29T10:51:10.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.680+0000 7f0288f86640 1 Processor -- start 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.681+0000 7f0288f86640 1 -- start start 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.681+0000 7f0288f86640 1 librados: setting wanted keys 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.681+0000 7f0288f86640 1 librados: calling monclient init 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.681+0000 7f0288f86640 10 monclient: init 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient(hunting): picked mon.noname-a con 0x7f02840b2800 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient(hunting): picked mon.noname-c con 0x7f028413c040 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient(hunting): picked mon.noname-b con 0x7f0284145330 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0284137700 con 0x7f02840b2800 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0284145d30 con 0x7f0284145330 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02841460f0 con 0x7f028413c040 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0288f86640 10 monclient(hunting): authenticate will time out at 1189.974983s 2024-03-29T10:51:10.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0283fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f0284145330 0x7f0284136ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:43326/0 (socket says 172.21.15.79:43326) 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.682+0000 7f0283fff640 1 -- 172.21.15.79:0/2927472282 learned_addr learned my addr 172.21.15.79:0/2927472282 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 client.?.objecter ms_handle_connect 0x7f0284145330 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 client.?.objecter ms_handle_connect 0x7f02840b2800 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 client.?.objecter ms_handle_connect 0x7f028413c040 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279733637 0 0) 0x7f0284137700 con 0x7f02840b2800 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0264004eb0 con 0x7f02840b2800 2024-03-29T10:51:10.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708122289 0 0) 0x7f0284145d30 con 0x7f0284145330 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0284137700 con 0x7f0284145330 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.683+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347389052 0 0) 0x7f02841460f0 con 0x7f028413c040 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0284145d30 con 0x7f028413c040 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2167126754 0 0) 0x7f0284137700 con 0x7f0284145330 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02841460f0 con 0x7f0284145330 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f027c001b90 con 0x7f0284145330 2024-03-29T10:51:10.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:10.675 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3635065143 0 0) 0x7f0284145d30 con 0x7f028413c040 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0284137700 con 0x7f028413c040 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1919261057 0 0) 0x7f02841460f0 con 0x7f0284145330 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 >> v1:172.21.15.79:6790/0 conn(0x7f028413c040 legacy=0x7f02841368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 >> v1:172.21.15.79:6789/0 conn(0x7f02840b2800 legacy=0x7f0284143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 monclient: found mon.b 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0284146280 con 0x7f0284145330 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0280ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0288f86640 5 monclient: authenticate success, global_id 4333 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0288f86640 10 monclient: _renew_subs 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0288f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:10.676 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.684+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0284144e50 con 0x7f0284145330 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.685+0000 7f0288f86640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.685+0000 7f0288f86640 10 monclient: _renew_subs 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.685+0000 7f0288f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.685+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0284143b00 con 0x7f0284145330 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.685+0000 7f0288f86640 1 librados: init done 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f0288f86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f0288f86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f0288f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0284145140 con 0x7f0284145330 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f027c002080 con 0x7f0284145330 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f0280ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.686+0000 7f02827fc640 4 set_mon_vals no callback set 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.687+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f027c004d00 con 0x7f0284145330 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.688+0000 7f0280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:10.677 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.688+0000 7f0280ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.688+0000 7f0280ff9640 10 monclient: dump: 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.688+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f027c01acd0 con 0x7f0284145330 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f02837fe640 10 monclient: get_auth_request con 0x7f026406df30 auth_method 0 2024-03-29T10:51:10.678 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f027c084ca0 con 0x7f0284145330 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f0280ff9640 10 client.4333.objecter ms_dispatch 0x7f0284000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f0280ff9640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f0280ff9640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f0280ff9640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.689+0000 7f0280ff9640 10 client.4333.objecter ms_handle_connect 0x7f026406df30 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.694+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f027c050430 con 0x7f0284145330 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.695+0000 7f0280ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:10.679 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:10.695+0000 7f0280ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:11.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.091+0000 7f0288f86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:11.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.091+0000 7f0288f86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:11.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.091+0000 7f0288f86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:11.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.091+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f028404f730 con 0x7f0284145330 2024-03-29T10:51:11.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.093+0000 7f0280ff9640 1 -- 172.21.15.79:0/2927472282 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f027c052530 con 0x7f0284145330 2024-03-29T10:51:11.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.093+0000 7f0280ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:11.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.093+0000 7f0280ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:11.075 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:11.075 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:11.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 10 librados: watch_flush enter 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 10 librados: watch_flush exit 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 10 client.4333.objecter successfully canceled tick 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 >> v1:172.21.15.79:6816/4290694627 conn(0x7f026406df30 legacy=0x7f02640703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 10 monclient: shutdown 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 >> v1:172.21.15.82:6789/0 conn(0x7f0284145330 legacy=0x7f0284136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 shutdown_connections 2024-03-29T10:51:11.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.097+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 >> 172.21.15.79:0/2927472282 conn(0x7f028406d0e0 msgr2=0x7f028413b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:11.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.098+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 shutdown_connections 2024-03-29T10:51:11.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.098+0000 7f0288f86640 1 -- 172.21.15.79:0/2927472282 wait complete. 2024-03-29T10:51:11.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:11.098+0000 7f0288f86640 1 librados: shutdown 2024-03-29T10:51:11.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:17.902 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.071+0000 7f45a4a63640 10 monclient: get_monmap_and_config 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.071+0000 7f45a4a63640 10 monclient: build_initial_monmap 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.071+0000 7f45a4a63640 10 monclient: monmap: 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:18.072329+0000 2024-03-29T10:51:18.053 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:18.072329+0000 2024-03-29T10:51:18.054 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:18.054 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.054 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:18.054 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:18.054 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:18.054 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.072+0000 7f45a4a63640 1 Processor -- start 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.072+0000 7f45a4a63640 1 -- start start 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.072+0000 7f45a4a63640 10 monclient: init 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient(hunting): picked mon.noname-a con 0x7f45a01457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient(hunting): picked mon.noname-c con 0x7f45a013f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient(hunting): picked mon.noname-b con 0x7f45a00a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:18.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a00aaef0 con 0x7f45a01457f0 2024-03-29T10:51:18.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a0145bd0 con 0x7f45a013f650 2024-03-29T10:51:18.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a0145eb0 con 0x7f45a00a9d70 2024-03-29T10:51:18.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:18.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f45a4a63640 10 monclient(hunting): authenticate will time out at 1197.365700s 2024-03-29T10:51:18.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f459dd9b640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f45a013f650 0x7f45a00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:55000/0 (socket says 172.21.15.79:55000) 2024-03-29T10:51:18.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.073+0000 7f459dd9b640 1 -- 172.21.15.79:0/2810620754 learned_addr learned my addr 172.21.15.79:0/2810620754 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198753670 0 0) 0x7f45a0145bd0 con 0x7f45a013f650 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f458c003520 con 0x7f45a013f650 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660256268 0 0) 0x7f45a0145eb0 con 0x7f45a00a9d70 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:18.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a0145bd0 con 0x7f45a00a9d70 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909938466 0 0) 0x7f458c003520 con 0x7f45a013f650 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 >> v1:172.21.15.79:6790/0 conn(0x7f45a00a9d70 legacy=0x7f45a014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 >> v1:172.21.15.79:6789/0 conn(0x7f45a01457f0 legacy=0x7f45a013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 monclient: found mon.noname-c 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0146340 con 0x7f45a013f650 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: _finish_auth 0 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:51:18.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4590001a10 con 0x7f45a013f650 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.074+0000 7f459d59a640 10 monclient: dump: 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:18.059 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f45a4a63640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f45a4a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4590001b70 con 0x7f45a013f650 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f459d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f459d59a640 1 -- 172.21.15.79:0/2810620754 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4590001f80 con 0x7f45a013f650 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f45a4a63640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f45a4a63640 4 set_mon_vals no callback set 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f459d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f459d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.075+0000 7f459d59a640 10 monclient: dump: 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:18.060 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:18.061 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 10 monclient: shutdown 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 1 -- 172.21.15.79:0/2810620754 >> v1:172.21.15.82:6789/0 conn(0x7f45a013f650 legacy=0x7f45a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 1 -- 172.21.15.79:0/2810620754 shutdown_connections 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 1 -- 172.21.15.79:0/2810620754 >> 172.21.15.79:0/2810620754 conn(0x7f45a0066a00 msgr2=0x7f45a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 1 -- 172.21.15.79:0/2810620754 shutdown_connections 2024-03-29T10:51:18.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.076+0000 7f45a4a63640 1 -- 172.21.15.79:0/2810620754 wait complete. 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 10 monclient: build_initial_monmap 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 10 monclient: monmap: 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 1 librados: starting msgr at 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 1 librados: starting objecter 2024-03-29T10:51:18.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 1 Processor -- start 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 1 -- start start 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 1 librados: setting wanted keys 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 1 librados: calling monclient init 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.078+0000 7f45a4a63640 10 monclient: init 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.079+0000 7f45a4a63640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.079+0000 7f45a4a63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.079+0000 7f45a4a63640 10 monclient(hunting): picked mon.noname-b con 0x7f45a00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 10 monclient(hunting): picked mon.noname-a con 0x7f45a013f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 10 monclient(hunting): picked mon.noname-c con 0x7f45a01457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a0139ef0 con 0x7f45a013f650 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f459e59c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f45a00a9d70 0x7f45a0137810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:56816/0 (socket says 172.21.15.79:56816) 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f459e59c640 1 -- 172.21.15.79:0/2372926416 learned_addr learned my addr 172.21.15.79:0/2372926416 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a0144c30 con 0x7f45a00a9d70 2024-03-29T10:51:18.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a0145180 con 0x7f45a01457f0 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f45a4a63640 10 monclient(hunting): authenticate will time out at 1197.372356s 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a00a9d70 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028112545 0 0) 0x7f45a0144c30 con 0x7f45a00a9d70 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580004eb0 con 0x7f45a00a9d70 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a013f650 2024-03-29T10:51:18.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 client.?.objecter ms_handle_connect 0x7f45a01457f0 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.080+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632881151 0 0) 0x7f45a0139ef0 con 0x7f45a013f650 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a0144c30 con 0x7f45a013f650 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026054576 0 0) 0x7f4580004eb0 con 0x7f45a00a9d70 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45a0139ef0 con 0x7f45a00a9d70 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4598001930 con 0x7f45a00a9d70 2024-03-29T10:51:18.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:18.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434226511 0 0) 0x7f45a0145180 con 0x7f45a01457f0 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580004eb0 con 0x7f45a01457f0 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3886237788 0 0) 0x7f45a0144c30 con 0x7f45a013f650 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45a0145180 con 0x7f45a013f650 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4590001ec0 con 0x7f45a013f650 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient(hunting): dump: 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:18.073 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2321121857 0 0) 0x7f45a0139ef0 con 0x7f45a00a9d70 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 >> v1:172.21.15.79:6790/0 conn(0x7f45a01457f0 legacy=0x7f45a01df610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 >> v1:172.21.15.79:6789/0 conn(0x7f45a013f650 legacy=0x7f45a01def00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 monclient: found mon.b 2024-03-29T10:51:18.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.081+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0145310 con 0x7f45a00a9d70 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f4596ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f4596ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f4596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 5 monclient: authenticate success, global_id 4339 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 10 monclient: _renew_subs 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a01399d0 con 0x7f45a00a9d70 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 10 monclient: _renew_subs 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:18.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a01401f0 con 0x7f45a00a9d70 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.082+0000 7f45a4a63640 1 librados: init done 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4598001a90 con 0x7f45a00a9d70 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f4596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4598003870 con 0x7f45a00a9d70 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f4596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f4596ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f4596ffd640 10 monclient: dump: 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:18.076 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f459cd99640 4 set_mon_vals no callback set 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.083+0000 7f45a4a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.084+0000 7f45a4a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.084+0000 7f45a4a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.084+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a0141780 con 0x7f45a00a9d70 2024-03-29T10:51:18.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.085+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f45980039d0 con 0x7f45a00a9d70 2024-03-29T10:51:18.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.086+0000 7f459dd9b640 10 monclient: get_auth_request con 0x7f458006e000 auth_method 0 2024-03-29T10:51:18.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.087+0000 7f4596ffd640 10 client.4339.objecter ms_handle_connect 0x7f458006e000 2024-03-29T10:51:18.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.087+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f4598084a00 con 0x7f45a00a9d70 2024-03-29T10:51:18.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.087+0000 7f4596ffd640 10 client.4339.objecter ms_dispatch 0x7f45a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:18.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.087+0000 7f4596ffd640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:18.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.087+0000 7f4596ffd640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:18.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.087+0000 7f4596ffd640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T10:51:18.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.094+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4598050110 con 0x7f45a00a9d70 2024-03-29T10:51:18.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.094+0000 7f4596ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:18.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.094+0000 7f4596ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:18.432 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.451+0000 7f45a4a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:18.432 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.451+0000 7f45a4a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:18.432 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.451+0000 7f45a4a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:18.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.451+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45a0075f10 con 0x7f45a00a9d70 2024-03-29T10:51:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.454+0000 7f4596ffd640 1 -- 172.21.15.79:0/2372926416 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f4598052210 con 0x7f45a00a9d70 2024-03-29T10:51:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.454+0000 7f4596ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.454+0000 7f4596ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:18.437 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:18.437 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 10 librados: watch_flush enter 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 10 librados: watch_flush exit 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 10 client.4339.objecter successfully canceled tick 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 >> v1:172.21.15.79:6816/4290694627 conn(0x7f458006e000 legacy=0x7f45800704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 10 monclient: shutdown 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 >> v1:172.21.15.82:6789/0 conn(0x7f45a00a9d70 legacy=0x7f45a0137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 shutdown_connections 2024-03-29T10:51:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 >> 172.21.15.79:0/2372926416 conn(0x7f45a0066a00 msgr2=0x7f45a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 shutdown_connections 2024-03-29T10:51:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 -- 172.21.15.79:0/2372926416 wait complete. 2024-03-29T10:51:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:18.458+0000 7f45a4a63640 1 librados: shutdown 2024-03-29T10:51:18.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:25.261 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:25.403 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.421+0000 7f178bbd0640 10 monclient: get_monmap_and_config 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.421+0000 7f178bbd0640 10 monclient: build_initial_monmap 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.421+0000 7f178bbd0640 10 monclient: monmap: 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:25.423148+0000 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:25.423148+0000 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:25.404 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:25.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.422+0000 7f178bbd0640 1 Processor -- start 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.422+0000 7f178bbd0640 1 -- start start 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.422+0000 7f178bbd0640 10 monclient: init 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.422+0000 7f178bbd0640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.422+0000 7f178bbd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.422+0000 7f178bbd0640 10 monclient(hunting): picked mon.noname-c con 0x7f17841457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 10 monclient(hunting): picked mon.noname-a con 0x7f178413f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 10 monclient(hunting): picked mon.noname-b con 0x7f17840a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17840aaef0 con 0x7f178413f650 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1784145bd0 con 0x7f17841457f0 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1784145eb0 con 0x7f17840a9d70 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:25.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178bbd0640 10 monclient(hunting): authenticate will time out at 1204.716182s 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178996c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f17841457f0 0x7f178413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:37018/0 (socket says 172.21.15.79:37018) 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178a16d640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f17840a9d70 0x7f178414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:58444/0 (socket says 172.21.15.79:58444) 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178a16d640 1 -- 172.21.15.79:0/920662213 learned_addr learned my addr 172.21.15.79:0/920662213 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557784864 0 0) 0x7f1784145eb0 con 0x7f17840a9d70 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1770003520 con 0x7f17840a9d70 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801451814 0 0) 0x7f1784145bd0 con 0x7f17841457f0 2024-03-29T10:51:25.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): _negotiate 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.423+0000 7f178896a640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.424+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1784145eb0 con 0x7f17841457f0 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.424+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405534429 0 0) 0x7f1770003520 con 0x7f17840a9d70 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.424+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 >> v1:172.21.15.82:6789/0 conn(0x7f17841457f0 legacy=0x7f178413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:25.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.424+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 >> v1:172.21.15.79:6789/0 conn(0x7f178413f650 legacy=0x7f17840b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: _finish_hunting 0 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 1 monclient: found mon.noname-b 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1784146340 con 0x7f17840a9d70 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: _finish_auth 0 2024-03-29T10:51:25.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: _check_auth_tickets 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1774001690 con 0x7f17840a9d70 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: dump: 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:25.411 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178bbd0640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178bbd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17740017f0 con 0x7f17840a9d70 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 1 -- 172.21.15.79:0/920662213 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1774001c00 con 0x7f17840a9d70 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178bbd0640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178bbd0640 4 set_mon_vals no callback set 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.426+0000 7f178896a640 10 monclient: dump: 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.412 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.427+0000 7f178bbd0640 10 monclient: shutdown 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.427+0000 7f178bbd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:25.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.427+0000 7f178bbd0640 1 -- 172.21.15.79:0/920662213 >> v1:172.21.15.79:6790/0 conn(0x7f17840a9d70 legacy=0x7f178414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:25.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.428+0000 7f178bbd0640 1 -- 172.21.15.79:0/920662213 shutdown_connections 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.428+0000 7f178bbd0640 1 -- 172.21.15.79:0/920662213 >> 172.21.15.79:0/920662213 conn(0x7f1784066a00 msgr2=0x7f1784066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.428+0000 7f178bbd0640 1 -- 172.21.15.79:0/920662213 shutdown_connections 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.428+0000 7f178bbd0640 1 -- 172.21.15.79:0/920662213 wait complete. 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 10 monclient: build_initial_monmap 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 10 monclient: monmap: 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:25.416 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 1 librados: starting msgr at 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 1 librados: starting objecter 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 1 Processor -- start 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 1 -- start start 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 1 librados: setting wanted keys 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 1 librados: calling monclient init 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.429+0000 7f178bbd0640 10 monclient: init 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient(hunting): picked mon.noname-c con 0x7f17840a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:25.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient(hunting): picked mon.noname-a con 0x7f178413f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient(hunting): picked mon.noname-b con 0x7f17841457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1784142110 con 0x7f178413f650 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17840ab600 con 0x7f17841457f0 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1784142d30 con 0x7f17840a9d70 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178bbd0640 10 monclient(hunting): authenticate will time out at 1204.723657s 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178996c640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f17840a9d70 0x7f1784142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:58914/0 (socket says 172.21.15.79:58914) 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f178996c640 1 -- 172.21.15.79:0/2713964327 learned_addr learned my addr 172.21.15.79:0/2713964327 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 10 client.?.objecter ms_handle_connect 0x7f17840a9d70 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516433207 0 0) 0x7f1784142d30 con 0x7f17840a9d70 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:25.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.430+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1768004eb0 con 0x7f17840a9d70 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 10 client.?.objecter ms_handle_connect 0x7f17841457f0 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748106025 0 0) 0x7f1768004eb0 con 0x7f17840a9d70 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1784142d30 con 0x7f17840a9d70 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f177c001cb0 con 0x7f17840a9d70 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 10 monclient(hunting): dump: 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.419 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3795658585 0 0) 0x7f1784142d30 con 0x7f17840a9d70 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 >> v1:172.21.15.82:6789/0 conn(0x7f17841457f0 legacy=0x7f17841e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:25.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.431+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 >> v1:172.21.15.79:6789/0 conn(0x7f178413f650 legacy=0x7f17841370f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 1 monclient: found mon.c 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1784140fd0 con 0x7f17840a9d70 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:25.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 10 monclient: _renew_subs 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178413fda0 con 0x7f17840a9d70 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 10 monclient: _renew_subs 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1784141cb0 con 0x7f17840a9d70 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f178bbd0640 1 librados: init done 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.433+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f177c004b60 con 0x7f17840a9d70 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.434+0000 7f177a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.434+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f177c004e70 con 0x7f17840a9d70 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.434+0000 7f177a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.434+0000 7f177a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:25.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.434+0000 7f177a7fc640 10 monclient: dump: 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.434+0000 7f177b7fe640 4 set_mon_vals no callback set 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.435+0000 7f178bbd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.435+0000 7f178bbd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.435+0000 7f178bbd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:25.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.435+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c005080 con 0x7f17840a9d70 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f177c01ae40 con 0x7f17840a9d70 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f177c084ee0 con 0x7f17840a9d70 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f177a7fc640 10 client.4382.objecter ms_dispatch 0x7f1784000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f177a7fc640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f177a7fc640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f178916b640 10 monclient: get_auth_request con 0x7f176806dc60 auth_method 0 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.436+0000 7f177a7fc640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T10:51:25.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.437+0000 7f177a7fc640 10 client.4382.objecter ms_handle_connect 0x7f176806dc60 2024-03-29T10:51:25.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.443+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f177c0505f0 con 0x7f17840a9d70 2024-03-29T10:51:25.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.443+0000 7f177a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:25.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.443+0000 7f177a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:25.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.752+0000 7f178bbd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.752+0000 7f178bbd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.752+0000 7f178bbd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.752+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176c005330 con 0x7f17840a9d70 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.753+0000 7f177a7fc640 1 -- 172.21.15.79:0/2713964327 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f177c0526f0 con 0x7f17840a9d70 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.753+0000 7f177a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.753+0000 7f177a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:25.736 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:25.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 10 librados: watch_flush enter 2024-03-29T10:51:25.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 10 librados: watch_flush exit 2024-03-29T10:51:25.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T10:51:25.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 10 client.4382.objecter successfully canceled tick 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 >> v1:172.21.15.79:6816/4290694627 conn(0x7f176806dc60 legacy=0x7f1768070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 10 monclient: shutdown 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 >> v1:172.21.15.79:6790/0 conn(0x7f17840a9d70 legacy=0x7f1784142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 shutdown_connections 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 >> 172.21.15.79:0/2713964327 conn(0x7f1784066a00 msgr2=0x7f178413aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 shutdown_connections 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 -- 172.21.15.79:0/2713964327 wait complete. 2024-03-29T10:51:25.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:25.756+0000 7f178bbd0640 1 librados: shutdown 2024-03-29T10:51:25.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:32.560 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:32.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.730+0000 7f286d7b6640 10 monclient: get_monmap_and_config 2024-03-29T10:51:32.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.730+0000 7f286d7b6640 10 monclient: build_initial_monmap 2024-03-29T10:51:32.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.730+0000 7f286d7b6640 10 monclient: monmap: 2024-03-29T10:51:32.712 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:32.712 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:32.731804+0000 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:32.731804+0000 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:32.713 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.734+0000 7f286d7b6640 1 Processor -- start 2024-03-29T10:51:32.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.734+0000 7f286d7b6640 1 -- start start 2024-03-29T10:51:32.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.734+0000 7f286d7b6640 10 monclient: init 2024-03-29T10:51:32.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:32.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:32.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f28680b0770 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:32.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f286814a700 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:32.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f28680ac5a0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:32.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680aba70 con 0x7f286814a700 2024-03-29T10:51:32.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f286d7b6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680aecb0 con 0x7f28680ac5a0 2024-03-29T10:51:32.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f2866ffd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f28680b0770 0x7f2868149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60198/0 (socket says 172.21.15.79:60198) 2024-03-29T10:51:32.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.735+0000 7f2866ffd640 1 -- 172.21.15.79:0/3529575071 learned_addr learned my addr 172.21.15.79:0/3529575071 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:32.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.738+0000 7f286d7b6640 1 -- 172.21.15.79:0/3529575071 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680af150 con 0x7f28680b0770 2024-03-29T10:51:32.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.738+0000 7f286d7b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:32.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.738+0000 7f286d7b6640 10 monclient(hunting): authenticate will time out at 1212.030931s 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419493569 0 0) 0x7f28680af150 con 0x7f28680b0770 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2850003520 con 0x7f28680b0770 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131687824 0 0) 0x7f28680aecb0 con 0x7f28680ac5a0 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:32.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:32.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:32.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:51:32.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.739+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28680af150 con 0x7f28680ac5a0 2024-03-29T10:51:32.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.740+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787977905 0 0) 0x7f2850003520 con 0x7f28680b0770 2024-03-29T10:51:32.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.740+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 >> v1:172.21.15.82:6789/0 conn(0x7f28680ac5a0 legacy=0x7f2868151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:32.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.740+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 >> v1:172.21.15.79:6789/0 conn(0x7f286814a700 legacy=0x7f28680abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:32.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:32.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 1 monclient: found mon.noname-b 2024-03-29T10:51:32.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:32.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28680ac9d0 con 0x7f28680b0770 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2858001690 con 0x7f28680b0770 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: dump: 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:32.724 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f286d7b6640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f286d7b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28580018b0 con 0x7f28680b0770 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 1 -- 172.21.15.79:0/3529575071 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2858001c40 con 0x7f28680b0770 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f2865ffb640 10 monclient: dump: 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.725 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.726 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:32.726 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.726 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:32.726 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:32.726 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:32.726 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f286d7b6640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:32.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.741+0000 7f286d7b6640 4 set_mon_vals no callback set 2024-03-29T10:51:32.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.742+0000 7f286d7b6640 10 monclient: shutdown 2024-03-29T10:51:32.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.742+0000 7f286d7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:32.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.742+0000 7f286d7b6640 1 -- 172.21.15.79:0/3529575071 >> v1:172.21.15.79:6790/0 conn(0x7f28680b0770 legacy=0x7f2868149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:32.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.742+0000 7f286d7b6640 1 -- 172.21.15.79:0/3529575071 shutdown_connections 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.742+0000 7f286d7b6640 1 -- 172.21.15.79:0/3529575071 >> 172.21.15.79:0/3529575071 conn(0x7f286806b660 msgr2=0x7f286806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.744+0000 7f286d7b6640 1 -- 172.21.15.79:0/3529575071 shutdown_connections 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.745+0000 7f286d7b6640 1 -- 172.21.15.79:0/3529575071 wait complete. 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.746+0000 7f286d7b6640 10 monclient: build_initial_monmap 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.746+0000 7f286d7b6640 10 monclient: monmap: 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:32.728 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.746+0000 7f286d7b6640 1 librados: starting msgr at 2024-03-29T10:51:32.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.746+0000 7f286d7b6640 1 librados: starting objecter 2024-03-29T10:51:32.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.747+0000 7f286d7b6640 1 Processor -- start 2024-03-29T10:51:32.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.747+0000 7f286d7b6640 1 -- start start 2024-03-29T10:51:32.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.747+0000 7f286d7b6640 1 librados: setting wanted keys 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.747+0000 7f286d7b6640 1 librados: calling monclient init 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.747+0000 7f286d7b6640 10 monclient: init 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.747+0000 7f286d7b6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f28680ac5a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f28680b0770 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f286814a700 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868125440 con 0x7f28680ac5a0 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868128240 con 0x7f28680b0770 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868129100 con 0x7f286814a700 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.748+0000 7f286d7b6640 10 monclient(hunting): authenticate will time out at 1212.040214s 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f28677fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f286814a700 0x7f28681e6bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60684/0 (socket says 172.21.15.79:60684) 2024-03-29T10:51:32.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f28677fe640 1 -- 172.21.15.79:0/3007405778 learned_addr learned my addr 172.21.15.79:0/3007405778 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 10 client.?.objecter ms_handle_connect 0x7f286814a700 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199396232 0 0) 0x7f2868129100 con 0x7f286814a700 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.750+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2854004eb0 con 0x7f286814a700 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 client.?.objecter ms_handle_connect 0x7f28680b0770 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909333368 0 0) 0x7f2854004eb0 con 0x7f286814a700 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2868129100 con 0x7f286814a700 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f285c001970 con 0x7f286814a700 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:32.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): dump: 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261338034 0 0) 0x7f2860001370 con 0x7f28680b0770 2024-03-29T10:51:32.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.752+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2854004eb0 con 0x7f28680b0770 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 client.?.objecter ms_handle_connect 0x7f28680ac5a0 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451113744 0 0) 0x7f2868125440 con 0x7f28680ac5a0 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2860001370 con 0x7f28680ac5a0 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736629123 0 0) 0x7f2854004eb0 con 0x7f28680b0770 2024-03-29T10:51:32.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2868125440 con 0x7f28680b0770 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2860001a20 con 0x7f28680b0770 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 10 monclient(hunting): dump: 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:32.743 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.753+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 830601384 0 0) 0x7f2860001370 con 0x7f28680ac5a0 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2854004eb0 con 0x7f28680ac5a0 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2858001a20 con 0x7f28680ac5a0 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient(hunting): dump: 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:32.744 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2353181623 0 0) 0x7f2868125440 con 0x7f28680b0770 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 >> v1:172.21.15.79:6790/0 conn(0x7f286814a700 legacy=0x7f28681e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 >> v1:172.21.15.79:6789/0 conn(0x7f28680ac5a0 legacy=0x7f2868146d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 monclient: found mon.b 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868147a30 con 0x7f28680b0770 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:32.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2860001f10 con 0x7f28680b0770 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2860003b10 con 0x7f28680b0770 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.754+0000 7f284f7fe640 10 monclient: dump: 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:32.746 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f28657fa640 4 set_mon_vals no callback set 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 5 monclient: authenticate success, global_id 4351 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 10 monclient: _renew_subs 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2868147c20 con 0x7f28680b0770 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 10 monclient: _renew_subs 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2868128850 con 0x7f28680b0770 2024-03-29T10:51:32.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.755+0000 7f286d7b6640 1 librados: init done 2024-03-29T10:51:32.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.759+0000 7f286d7b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.759+0000 7f286d7b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.759+0000 7f286d7b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.759+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28680ab560 con 0x7f28680b0770 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f2860019ae0 con 0x7f28680b0770 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f28600838c0 con 0x7f28680b0770 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f284f7fe640 10 client.4351.objecter ms_dispatch 0x7f2868000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f284f7fe640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f284f7fe640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f284f7fe640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.760+0000 7f2866ffd640 10 monclient: get_auth_request con 0x7f285406e200 auth_method 0 2024-03-29T10:51:32.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.761+0000 7f284f7fe640 10 client.4351.objecter ms_handle_connect 0x7f285406e200 2024-03-29T10:51:32.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.768+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f286004f1a0 con 0x7f28680b0770 2024-03-29T10:51:32.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.768+0000 7f284f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:32.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:32.768+0000 7f284f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:33.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.099+0000 7f286d7b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:33.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.099+0000 7f286d7b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:33.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.099+0000 7f286d7b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:51:33.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.099+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2868075f10 con 0x7f28680b0770 2024-03-29T10:51:33.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.101+0000 7f284f7fe640 1 -- 172.21.15.79:0/3007405778 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f28600510d0 con 0x7f28680b0770 2024-03-29T10:51:33.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.101+0000 7f284f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:33.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.101+0000 7f284f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:33.083 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:33.083 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:33.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 10 librados: watch_flush enter 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 10 librados: watch_flush exit 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 10 client.4351.objecter successfully canceled tick 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 >> v1:172.21.15.79:6816/4290694627 conn(0x7f285406e200 legacy=0x7f28540706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 10 monclient: shutdown 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:33.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.104+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 >> v1:172.21.15.82:6789/0 conn(0x7f28680b0770 legacy=0x7f28681db6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:33.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.107+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 shutdown_connections 2024-03-29T10:51:33.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.107+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 >> 172.21.15.79:0/3007405778 conn(0x7f286806b660 msgr2=0x7f28680b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:33.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.107+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 shutdown_connections 2024-03-29T10:51:33.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.107+0000 7f286d7b6640 1 -- 172.21.15.79:0/3007405778 wait complete. 2024-03-29T10:51:33.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:33.108+0000 7f286d7b6640 1 librados: shutdown 2024-03-29T10:51:33.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:39.914 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.082+0000 7ff537d1c640 10 monclient: get_monmap_and_config 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.082+0000 7ff537d1c640 10 monclient: build_initial_monmap 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.082+0000 7ff537d1c640 10 monclient: monmap: 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:40.083226+0000 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:40.083226+0000 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:40.064 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:40.065 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:40.065 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:40.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.084+0000 7ff537d1c640 1 Processor -- start 2024-03-29T10:51:40.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.084+0000 7ff537d1c640 1 -- start start 2024-03-29T10:51:40.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.084+0000 7ff537d1c640 10 monclient: init 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff5300b4790 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5300aea90 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff5300b39f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5300b0c60 con 0x7ff5300b39f0 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5300af020 con 0x7ff5300aea90 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53013fbd0 con 0x7ff5300b4790 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff537d1c640 10 monclient(hunting): authenticate will time out at 1219.377877s 2024-03-29T10:51:40.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff535ab8640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7ff5300b4790 0x7ff5300b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:57136/0 (socket says 172.21.15.79:57136) 2024-03-29T10:51:40.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.085+0000 7ff535ab8640 1 -- 172.21.15.79:0/767958023 learned_addr learned my addr 172.21.15.79:0/767958023 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:40.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.088+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294655833 0 0) 0x7ff53013fbd0 con 0x7ff5300b4790 2024-03-29T10:51:40.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.088+0000 7ff534ab6640 10 monclient(hunting): _negotiate 2024-03-29T10:51:40.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.088+0000 7ff534ab6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:40.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.088+0000 7ff534ab6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:40.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.088+0000 7ff534ab6640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:51:40.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.088+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff51c003520 con 0x7ff5300b4790 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3612458969 0 0) 0x7ff51c003520 con 0x7ff5300b4790 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 >> v1:172.21.15.82:6789/0 conn(0x7ff5300aea90 legacy=0x7ff5300af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 >> v1:172.21.15.79:6789/0 conn(0x7ff5300b39f0 legacy=0x7ff53013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: _finish_hunting 0 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 monclient: found mon.noname-b 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53013fd60 con 0x7ff5300b4790 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: _finish_auth 0 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: _check_auth_tickets 2024-03-29T10:51:40.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff520001690 con 0x7ff5300b4790 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: dump: 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff537d1c640 5 monclient: authenticate success, global_id 4391 2024-03-29T10:51:40.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff537d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff520001860 con 0x7ff5300b4790 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff534ab6640 1 -- 172.21.15.79:0/767958023 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff520001c70 con 0x7ff5300b4790 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff537d1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.090+0000 7ff537d1c640 4 set_mon_vals no callback set 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.091+0000 7ff534ab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.091+0000 7ff534ab6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.091+0000 7ff534ab6640 10 monclient: dump: 2024-03-29T10:51:40.075 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.091+0000 7ff537d1c640 10 monclient: shutdown 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.091+0000 7ff537d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.091+0000 7ff537d1c640 1 -- 172.21.15.79:0/767958023 >> v1:172.21.15.79:6790/0 conn(0x7ff5300b4790 legacy=0x7ff5300b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.092+0000 7ff537d1c640 1 -- 172.21.15.79:0/767958023 shutdown_connections 2024-03-29T10:51:40.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.092+0000 7ff537d1c640 1 -- 172.21.15.79:0/767958023 >> 172.21.15.79:0/767958023 conn(0x7ff530066a00 msgr2=0x7ff530066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:40.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.094+0000 7ff537d1c640 1 -- 172.21.15.79:0/767958023 shutdown_connections 2024-03-29T10:51:40.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.095+0000 7ff537d1c640 1 -- 172.21.15.79:0/767958023 wait complete. 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.095+0000 7ff537d1c640 10 monclient: build_initial_monmap 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.095+0000 7ff537d1c640 10 monclient: monmap: 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:40.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.096+0000 7ff537d1c640 1 librados: starting msgr at 2024-03-29T10:51:40.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.096+0000 7ff537d1c640 1 librados: starting objecter 2024-03-29T10:51:40.079 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.096+0000 7ff537d1c640 1 Processor -- start 2024-03-29T10:51:40.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.098+0000 7ff537d1c640 1 -- start start 2024-03-29T10:51:40.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.098+0000 7ff537d1c640 1 librados: setting wanted keys 2024-03-29T10:51:40.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.098+0000 7ff537d1c640 1 librados: calling monclient init 2024-03-29T10:51:40.080 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.098+0000 7ff537d1c640 10 monclient: init 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5300b39f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff5300b4790 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff530133de0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5301e40a0 con 0x7ff5300b4790 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5301336e0 con 0x7ff530133de0 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff530131360 con 0x7ff5300b39f0 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:40.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.099+0000 7ff537d1c640 10 monclient(hunting): authenticate will time out at 1219.391887s 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.100+0000 7ff535ab8640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7ff5300b39f0 0x7ff530132100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:57884/0 (socket says 172.21.15.79:57884) 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.100+0000 7ff535ab8640 1 -- 172.21.15.79:0/1911185495 learned_addr learned my addr 172.21.15.79:0/1911185495 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.100+0000 7ff5267fc640 10 client.?.objecter ms_handle_connect 0x7ff5300b39f0 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.100+0000 7ff5267fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702758176 0 0) 0x7ff530131360 con 0x7ff5300b39f0 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:51:40.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff514004eb0 con 0x7ff5300b39f0 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215251443 0 0) 0x7ff514004eb0 con 0x7ff5300b39f0 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.101+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff530131360 con 0x7ff5300b39f0 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff520001c70 con 0x7ff5300b39f0 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 10 monclient(hunting): dump: 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:40.085 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:40.086 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:40.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:40.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2214994014 0 0) 0x7ff530131360 con 0x7ff5300b39f0 2024-03-29T10:51:40.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 >> v1:172.21.15.82:6789/0 conn(0x7ff530133de0 legacy=0x7ff5301e32d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:40.086 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.102+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 >> v1:172.21.15.79:6789/0 conn(0x7ff5300b4790 legacy=0x7ff5301e4550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:40.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.103+0000 7ff5267fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:40.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.103+0000 7ff5267fc640 1 monclient: found mon.c 2024-03-29T10:51:40.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:40.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:40.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff530135590 con 0x7ff5300b39f0 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 10 monclient: _renew_subs 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff530132810 con 0x7ff5300b39f0 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 10 monclient: _renew_subs 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5301e4e40 con 0x7ff5300b39f0 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff537d1c640 1 librados: init done 2024-03-29T10:51:40.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff520004a90 con 0x7ff5300b39f0 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff520004ee0 con 0x7ff5300b39f0 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.104+0000 7ff5277fe640 4 set_mon_vals no callback set 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.105+0000 7ff5267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.105+0000 7ff5267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.105+0000 7ff5267fc640 10 monclient: dump: 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:40.089 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:40.090 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.106+0000 7ff537d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.106+0000 7ff537d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.106+0000 7ff537d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.106+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff518005080 con 0x7ff5300b39f0 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.107+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7ff520005040 con 0x7ff5300b39f0 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.107+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7ff520084c50 con 0x7ff5300b39f0 2024-03-29T10:51:40.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.107+0000 7ff5267fc640 10 client.4394.objecter ms_dispatch 0x7ff530000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:40.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.108+0000 7ff5267fc640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:40.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.108+0000 7ff5267fc640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:40.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.108+0000 7ff5267fc640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T10:51:40.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.109+0000 7ff5352b7640 10 monclient: get_auth_request con 0x7ff51406dc70 auth_method 0 2024-03-29T10:51:40.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.110+0000 7ff5267fc640 10 client.4394.objecter ms_handle_connect 0x7ff51406dc70 2024-03-29T10:51:40.096 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.115+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff520050530 con 0x7ff5300b39f0 2024-03-29T10:51:40.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.115+0000 7ff5267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:40.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.115+0000 7ff5267fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.449+0000 7ff537d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.449+0000 7ff537d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.449+0000 7ff537d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.449+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff518005330 con 0x7ff5300b39f0 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.450+0000 7ff5267fc640 1 -- 172.21.15.79:0/1911185495 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7ff520052460 con 0x7ff5300b39f0 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.450+0000 7ff5267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:40.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.450+0000 7ff5267fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:40.432 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:40.432 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 10 librados: watch_flush enter 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 10 librados: watch_flush exit 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 10 client.4394.objecter successfully canceled tick 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 >> v1:172.21.15.79:6816/4290694627 conn(0x7ff51406dc70 legacy=0x7ff514070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 10 monclient: shutdown 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:40.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.453+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 >> v1:172.21.15.79:6790/0 conn(0x7ff5300b39f0 legacy=0x7ff530132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:40.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.454+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 shutdown_connections 2024-03-29T10:51:40.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.455+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 >> 172.21.15.79:0/1911185495 conn(0x7ff530066a00 msgr2=0x7ff5300b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:40.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.455+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 shutdown_connections 2024-03-29T10:51:40.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.455+0000 7ff537d1c640 1 -- 172.21.15.79:0/1911185495 wait complete. 2024-03-29T10:51:40.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:40.455+0000 7ff537d1c640 1 librados: shutdown 2024-03-29T10:51:40.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:47.260 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.422+0000 7f38a2281640 10 monclient: get_monmap_and_config 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.422+0000 7f38a2281640 10 monclient: build_initial_monmap 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.422+0000 7f38a2281640 10 monclient: monmap: 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:47.423665+0000 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:47.423665+0000 2024-03-29T10:51:47.404 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:47.405 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:47.405 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:47.405 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:47.405 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:47.405 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.424+0000 7f38a2281640 1 Processor -- start 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.424+0000 7f38a2281640 1 -- start start 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.424+0000 7f38a2281640 10 monclient: init 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.424+0000 7f38a2281640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 10 monclient(hunting): picked mon.noname-a con 0x7f389c0b4330 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 10 monclient(hunting): picked mon.noname-c con 0x7f389c1470b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:47.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 10 monclient(hunting): picked mon.noname-b con 0x7f389c1439b0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c06f2a0 con 0x7f389c0b4330 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c146b90 con 0x7f389c1470b0 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c0b2ca0 con 0x7f389c1439b0 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.425+0000 7f38a2281640 10 monclient(hunting): authenticate will time out at 1226.717208s 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.427+0000 7f389b7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f389c1470b0 0x7f389c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:43359/0 (socket says 172.21.15.79:43359) 2024-03-29T10:51:47.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.427+0000 7f389b7fe640 1 -- 172.21.15.79:0/3945286410 learned_addr learned my addr 172.21.15.79:0/3945286410 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:47.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163562123 0 0) 0x7f389c0b2ca0 con 0x7f389c1439b0 2024-03-29T10:51:47.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:47.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:47.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:47.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:51:47.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3880003520 con 0x7f389c1439b0 2024-03-29T10:51:47.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645749903 0 0) 0x7f389c146b90 con 0x7f389c1470b0 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c0b2ca0 con 0x7f389c1470b0 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863685390 0 0) 0x7f389c06f2a0 con 0x7f389c0b4330 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c146b90 con 0x7f389c0b4330 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245597825 0 0) 0x7f3880003520 con 0x7f389c1439b0 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 >> v1:172.21.15.82:6789/0 conn(0x7f389c1470b0 legacy=0x7f389c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 >> v1:172.21.15.79:6789/0 conn(0x7f389c0b4330 legacy=0x7f389c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 monclient: found mon.noname-b 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c06e0c0 con 0x7f389c1439b0 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: _finish_auth 0 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f38a2281640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f38a2281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f388c001690 con 0x7f389c1439b0 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:47.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: dump: 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f38a2281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f388c0018b0 con 0x7f389c1439b0 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f389affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f38a2281640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:47.413 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.428+0000 7f38a2281640 4 set_mon_vals no callback set 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f389affd640 1 -- 172.21.15.79:0/3945286410 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f388c001c40 con 0x7f389c1439b0 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f389affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f389affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f389affd640 10 monclient: dump: 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:47.414 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f38a2281640 10 monclient: shutdown 2024-03-29T10:51:47.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f38a2281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:47.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f38a2281640 1 -- 172.21.15.79:0/3945286410 >> v1:172.21.15.79:6790/0 conn(0x7f389c1439b0 legacy=0x7f389c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f38a2281640 1 -- 172.21.15.79:0/3945286410 shutdown_connections 2024-03-29T10:51:47.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.429+0000 7f38a2281640 1 -- 172.21.15.79:0/3945286410 >> 172.21.15.79:0/3945286410 conn(0x7f389c066a00 msgr2=0x7f389c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:47.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.430+0000 7f38a2281640 1 -- 172.21.15.79:0/3945286410 shutdown_connections 2024-03-29T10:51:47.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.430+0000 7f38a2281640 1 -- 172.21.15.79:0/3945286410 wait complete. 2024-03-29T10:51:47.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.431+0000 7f38a2281640 10 monclient: build_initial_monmap 2024-03-29T10:51:47.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.431+0000 7f38a2281640 10 monclient: monmap: 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:47.421 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.431+0000 7f38a2281640 1 librados: starting msgr at 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.431+0000 7f38a2281640 1 librados: starting objecter 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.432+0000 7f38a2281640 1 Processor -- start 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 1 -- start start 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 1 librados: setting wanted keys 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 1 librados: calling monclient init 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient: init 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient(hunting): picked mon.noname-c con 0x7f389c0b4330 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient(hunting): picked mon.noname-a con 0x7f389c1439b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient(hunting): picked mon.noname-b con 0x7f389c1470b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:47.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c1db6b0 con 0x7f389c1439b0 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c127560 con 0x7f389c1470b0 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389c122850 con 0x7f389c0b4330 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.433+0000 7f38a2281640 10 monclient(hunting): authenticate will time out at 1226.725962s 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.435+0000 7f389bfff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f389c0b4330 0x7f389c1dbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:55010/0 (socket says 172.21.15.79:55010) 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.435+0000 7f389bfff640 1 -- 172.21.15.79:0/1530984034 learned_addr learned my addr 172.21.15.79:0/1530984034 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 client.?.objecter ms_handle_connect 0x7f389c0b4330 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723848917 0 0) 0x7f389c122850 con 0x7f389c0b4330 2024-03-29T10:51:47.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3884004eb0 con 0x7f389c0b4330 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 client.?.objecter ms_handle_connect 0x7f389c1439b0 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419951679 0 0) 0x7f389c1db6b0 con 0x7f389c1439b0 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:47.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:51:47.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.437+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c122850 con 0x7f389c1439b0 2024-03-29T10:51:47.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.438+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 676117465 0 0) 0x7f3884004eb0 con 0x7f389c0b4330 2024-03-29T10:51:47.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.438+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f389c1db6b0 con 0x7f389c0b4330 2024-03-29T10:51:47.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3894001cf0 con 0x7f389c0b4330 2024-03-29T10:51:47.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:47.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 18413691 0 0) 0x7f389c1db6b0 con 0x7f389c0b4330 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 >> v1:172.21.15.82:6789/0 conn(0x7f389c1470b0 legacy=0x7f389c125b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 >> v1:172.21.15.79:6789/0 conn(0x7f389c1439b0 legacy=0x7f389c1411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 1 monclient: found mon.c 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c1229e0 con 0x7f389c0b4330 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 client.?.objecter ms_handle_connect 0x7f389c1470b0 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 10 monclient: _renew_subs 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:47.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c1dc2a0 con 0x7f389c0b4330 2024-03-29T10:51:47.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:47.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 10 monclient: _renew_subs 2024-03-29T10:51:47.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:47.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c141aa0 con 0x7f389c0b4330 2024-03-29T10:51:47.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f38a2281640 1 librados: init done 2024-03-29T10:51:47.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.442+0000 7f3898ff9640 10 client.4400.objecter resend_mon_ops 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.445+0000 7f38a2281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.445+0000 7f38a2281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.445+0000 7f38a2281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.445+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c075f10 con 0x7f389c0b4330 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3894002050 con 0x7f389c0b4330 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f3898ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3894001ed0 con 0x7f389c0b4330 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f3898ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f3898ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:47.429 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f3898ff9640 10 monclient: dump: 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:47.430 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:47.431 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:47.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.446+0000 7f389a7fc640 4 set_mon_vals no callback set 2024-03-29T10:51:47.431 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.448+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f3894003c40 con 0x7f389c0b4330 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.450+0000 7f389b7fe640 10 monclient: get_auth_request con 0x7f388406dad0 auth_method 0 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.451+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f3894084bd0 con 0x7f389c0b4330 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.451+0000 7f3898ff9640 10 client.4400.objecter ms_dispatch 0x7f389c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.451+0000 7f3898ff9640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.451+0000 7f3898ff9640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.451+0000 7f3898ff9640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T10:51:47.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.451+0000 7f3898ff9640 10 client.4400.objecter ms_handle_connect 0x7f388406dad0 2024-03-29T10:51:47.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.459+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3894050510 con 0x7f389c0b4330 2024-03-29T10:51:47.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.459+0000 7f3898ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:47.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.459+0000 7f3898ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:47.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.788+0000 7f38a2281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:47.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.788+0000 7f38a2281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:47.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.788+0000 7f38a2281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:47.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.788+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f389c074f10 con 0x7f389c0b4330 2024-03-29T10:51:47.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.789+0000 7f3898ff9640 1 -- 172.21.15.79:0/1530984034 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f3894052440 con 0x7f389c0b4330 2024-03-29T10:51:47.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.789+0000 7f3898ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:47.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.789+0000 7f3898ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:47.773 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:47.773 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:47.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 10 librados: watch_flush enter 2024-03-29T10:51:47.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 10 librados: watch_flush exit 2024-03-29T10:51:47.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T10:51:47.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 10 client.4400.objecter successfully canceled tick 2024-03-29T10:51:47.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 >> v1:172.21.15.79:6816/4290694627 conn(0x7f388406dad0 legacy=0x7f388406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 10 monclient: shutdown 2024-03-29T10:51:47.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:47.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 >> v1:172.21.15.79:6790/0 conn(0x7f389c0b4330 legacy=0x7f389c1dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:47.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.793+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 shutdown_connections 2024-03-29T10:51:47.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.794+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 >> 172.21.15.79:0/1530984034 conn(0x7f389c066a00 msgr2=0x7f389c0b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:47.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.795+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 shutdown_connections 2024-03-29T10:51:47.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.795+0000 7f38a2281640 1 -- 172.21.15.79:0/1530984034 wait complete. 2024-03-29T10:51:47.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:47.795+0000 7f38a2281640 1 librados: shutdown 2024-03-29T10:51:47.793 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:54.600 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.758+0000 7f44f27a3640 10 monclient: get_monmap_and_config 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.758+0000 7f44f27a3640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.758+0000 7f44f27a3640 10 monclient: monmap: 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:51:54.759312+0000 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:51:54.759312+0000 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:51:54.740 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:54.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.759+0000 7f44f27a3640 1 Processor -- start 2024-03-29T10:51:54.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.759+0000 7f44f27a3640 1 -- start start 2024-03-29T10:51:54.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.759+0000 7f44f27a3640 10 monclient: init 2024-03-29T10:51:54.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.760+0000 7f44f27a3640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.760+0000 7f44f27a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:54.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.760+0000 7f44f27a3640 10 monclient(hunting): picked mon.noname-b con 0x7f44e40389e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:54.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.760+0000 7f44f27a3640 10 monclient(hunting): picked mon.noname-c con 0x7f44e4038dc0 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:54.745 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.763+0000 7f44f17a1640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f44e40389e0 0x7f44e4033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:56664/0 (socket says 172.21.15.79:56664) 2024-03-29T10:51:54.746 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f27a3640 10 monclient(hunting): picked mon.noname-a con 0x7f44e4030660 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:54.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f27a3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40c54c0 con 0x7f44e4030660 2024-03-29T10:51:54.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f27a3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e4036430 con 0x7f44e4038dc0 2024-03-29T10:51:54.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f27a3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e4039840 con 0x7f44e40389e0 2024-03-29T10:51:54.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f27a3640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f27a3640 10 monclient(hunting): authenticate will time out at 1234.057789s 2024-03-29T10:51:54.747 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.765+0000 7f44f17a1640 1 -- 172.21.15.79:0/901418189 learned_addr learned my addr 172.21.15.79:0/901418189 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831784670 0 0) 0x7f44e4039840 con 0x7f44e40389e0 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d8003520 con 0x7f44e40389e0 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4053163838 0 0) 0x7f44d8003520 con 0x7f44e40389e0 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 >> v1:172.21.15.82:6789/0 conn(0x7f44e4038dc0 legacy=0x7f44e40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:54.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.766+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 >> v1:172.21.15.79:6789/0 conn(0x7f44e4030660 legacy=0x7f44e40d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 1 monclient: found mon.noname-b 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4035690 con 0x7f44e40389e0 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.770+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f44e8001690 con 0x7f44e40389e0 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.771+0000 7f44e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.771+0000 7f44e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:54.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.771+0000 7f44e3fff640 10 monclient: dump: 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44f27a3640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44f27a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e8001860 con 0x7f44e40389e0 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44e3fff640 1 -- 172.21.15.79:0/901418189 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f44e8001cb0 con 0x7f44e40389e0 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44e3fff640 10 monclient: dump: 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44f27a3640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:54.758 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.773+0000 7f44f27a3640 4 set_mon_vals no callback set 2024-03-29T10:51:54.759 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.774+0000 7f44f27a3640 10 monclient: shutdown 2024-03-29T10:51:54.759 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.774+0000 7f44f27a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:54.759 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.774+0000 7f44f27a3640 1 -- 172.21.15.79:0/901418189 >> v1:172.21.15.79:6790/0 conn(0x7f44e40389e0 legacy=0x7f44e4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.759 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.774+0000 7f44f27a3640 1 -- 172.21.15.79:0/901418189 shutdown_connections 2024-03-29T10:51:54.759 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.775+0000 7f44f27a3640 1 -- 172.21.15.79:0/901418189 >> 172.21.15.79:0/901418189 conn(0x7f44e4005c00 msgr2=0x7f44e4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:54.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.775+0000 7f44f27a3640 1 -- 172.21.15.79:0/901418189 shutdown_connections 2024-03-29T10:51:54.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.775+0000 7f44f27a3640 1 -- 172.21.15.79:0/901418189 wait complete. 2024-03-29T10:51:54.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.776+0000 7f44f27a3640 10 monclient: build_initial_monmap 2024-03-29T10:51:54.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.776+0000 7f44f27a3640 10 monclient: monmap: 2024-03-29T10:51:54.760 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.776+0000 7f44f27a3640 1 librados: starting msgr at 2024-03-29T10:51:54.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.777+0000 7f44f27a3640 1 librados: starting objecter 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.777+0000 7f44f27a3640 1 Processor -- start 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.777+0000 7f44f27a3640 1 -- start start 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.777+0000 7f44f27a3640 1 librados: setting wanted keys 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.777+0000 7f44f27a3640 1 librados: calling monclient init 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.777+0000 7f44f27a3640 10 monclient: init 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.778+0000 7f44f27a3640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:54.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.778+0000 7f44f27a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 10 monclient(hunting): picked mon.noname-b con 0x7f44e4030660 addr v1:172.21.15.82:6789/0 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 10 monclient(hunting): picked mon.noname-c con 0x7f44e40389e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 10 monclient(hunting): picked mon.noname-a con 0x7f44e41609e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40c7170 con 0x7f44e41609e0 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e415fdd0 con 0x7f44e4030660 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e402ffb0 con 0x7f44e40389e0 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f0fa0640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f44e40389e0 0x7f44e40bfe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:58136/0 (socket says 172.21.15.79:58136) 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f0fa0640 1 -- 172.21.15.79:0/600900206 learned_addr learned my addr 172.21.15.79:0/600900206 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:54.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44e1ffb640 10 client.?.objecter ms_handle_connect 0x7f44e40389e0 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44f27a3640 10 monclient(hunting): authenticate will time out at 1234.073014s 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.780+0000 7f44e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176968823 0 0) 0x7f44e402ffb0 con 0x7f44e40389e0 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d0004eb0 con 0x7f44e40389e0 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861490489 0 0) 0x7f44d0004eb0 con 0x7f44e40389e0 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44e402ffb0 con 0x7f44e40389e0 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f44dc0017d0 con 0x7f44e40389e0 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:54.766 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3520244858 0 0) 0x7f44e402ffb0 con 0x7f44e40389e0 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 >> v1:172.21.15.82:6789/0 conn(0x7f44e4030660 legacy=0x7f44e41602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:54.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.781+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 >> v1:172.21.15.79:6789/0 conn(0x7f44e41609e0 legacy=0x7f44e40c0550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 1 monclient: found mon.c 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4030140 con 0x7f44e40389e0 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:54.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:54.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44f27a3640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44f27a3640 10 monclient: _renew_subs 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44f27a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.787+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e40c7590 con 0x7f44e40389e0 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.788+0000 7f44f27a3640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.788+0000 7f44f27a3640 10 monclient: _renew_subs 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.788+0000 7f44f27a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.788+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e402fb90 con 0x7f44e40389e0 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.788+0000 7f44f27a3640 1 librados: init done 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44dc001930 con 0x7f44e40389e0 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f44dc004a40 con 0x7f44e40389e0 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:54.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 10 monclient: dump: 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e2ffd640 4 set_mon_vals no callback set 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44f27a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.789+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f44dc004c60 con 0x7f44e40389e0 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.790+0000 7f44f27a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.790+0000 7f44f27a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.790+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44cc00d890 con 0x7f44e40389e0 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f44dc084a80 con 0x7f44e40389e0 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44e1ffb640 10 client.4406.objecter ms_dispatch 0x7f44ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44e1ffb640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44e1ffb640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44f17a1640 10 monclient: get_auth_request con 0x7f44d006dd50 auth_method 0 2024-03-29T10:51:54.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44e1ffb640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T10:51:54.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.791+0000 7f44e1ffb640 10 client.4406.objecter ms_handle_connect 0x7f44d006dd50 2024-03-29T10:51:54.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.798+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44dc050360 con 0x7f44e40389e0 2024-03-29T10:51:54.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.798+0000 7f44e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:54.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:54.798+0000 7f44e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:55.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.106+0000 7f44f27a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:55.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.106+0000 7f44f27a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:55.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.106+0000 7f44f27a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:51:55.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.106+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44cc00db40 con 0x7f44e40389e0 2024-03-29T10:51:55.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.107+0000 7f44e1ffb640 1 -- 172.21.15.79:0/600900206 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f44dc052290 con 0x7f44e40389e0 2024-03-29T10:51:55.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.107+0000 7f44e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:55.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.107+0000 7f44e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:55.089 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:51:55.089 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:55.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.109+0000 7f44f27a3640 10 librados: watch_flush enter 2024-03-29T10:51:55.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 10 librados: watch_flush exit 2024-03-29T10:51:55.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T10:51:55.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 10 client.4406.objecter successfully canceled tick 2024-03-29T10:51:55.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 >> v1:172.21.15.79:6816/4290694627 conn(0x7f44d006dd50 legacy=0x7f44d0070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 10 monclient: shutdown 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 >> v1:172.21.15.79:6790/0 conn(0x7f44e40389e0 legacy=0x7f44e40bfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 shutdown_connections 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 >> 172.21.15.79:0/600900206 conn(0x7f44e4005c00 msgr2=0x7f44e4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 shutdown_connections 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 -- 172.21.15.79:0/600900206 wait complete. 2024-03-29T10:51:55.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:51:55.110+0000 7f44f27a3640 1 librados: shutdown 2024-03-29T10:51:55.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:01.915 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:02.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.090+0000 7fe8e38e4640 10 monclient: get_monmap_and_config 2024-03-29T10:52:02.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.090+0000 7fe8e38e4640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.090+0000 7fe8e38e4640 10 monclient: monmap: 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:02.091814+0000 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:02.091814+0000 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:02.073 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.094+0000 7fe8e38e4640 1 Processor -- start 2024-03-29T10:52:02.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.094+0000 7fe8e38e4640 1 -- start start 2024-03-29T10:52:02.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.094+0000 7fe8e38e4640 10 monclient: init 2024-03-29T10:52:02.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:02.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d4034600 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:02.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d40349e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:02.077 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d402b590 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d4032f10 con 0x7fe8d4034600 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40be190 con 0x7fe8d402b590 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d402c740 con 0x7fe8d40349e0 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.095+0000 7fe8e38e4640 10 monclient(hunting): authenticate will time out at 1241.387965s 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.097+0000 7fe8e0e7f640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe8d40349e0 0x7fe8d40c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:52478/0 (socket says 172.21.15.79:52478) 2024-03-29T10:52:02.078 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.097+0000 7fe8e0e7f640 1 -- 172.21.15.79:0/2479689665 learned_addr learned my addr 172.21.15.79:0/2479689665 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:02.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.099+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371940412 0 0) 0x7fe8d402c740 con 0x7fe8d40349e0 2024-03-29T10:52:02.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.099+0000 7fe8d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.099+0000 7fe8d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.081 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.099+0000 7fe8d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.099+0000 7fe8d3fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.099+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c8003520 con 0x7fe8d40349e0 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808448558 0 0) 0x7fe8d40be190 con 0x7fe8d402b590 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d402c740 con 0x7fe8d402b590 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1459954399 0 0) 0x7fe8c8003520 con 0x7fe8d40349e0 2024-03-29T10:52:02.082 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 >> v1:172.21.15.82:6789/0 conn(0x7fe8d402b590 legacy=0x7fe8d40c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.083 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.100+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 >> v1:172.21.15.79:6789/0 conn(0x7fe8d4034600 legacy=0x7fe8d40bda80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 1 monclient: found mon.noname-b 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d402c9b0 con 0x7fe8d40349e0 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe8dc04f730 con 0x7fe8d40349e0 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: dump: 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:02.084 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8dc0008d0 con 0x7fe8d40349e0 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 1 -- 172.21.15.79:0/2479689665 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe8dc0758b0 con 0x7fe8d40349e0 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.101+0000 7fe8d3fff640 10 monclient: dump: 2024-03-29T10:52:02.085 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:02.086 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.103+0000 7fe8e38e4640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:52:02.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.103+0000 7fe8e38e4640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:02.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.103+0000 7fe8e38e4640 4 set_mon_vals no callback set 2024-03-29T10:52:02.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.104+0000 7fe8e38e4640 10 monclient: shutdown 2024-03-29T10:52:02.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.104+0000 7fe8e38e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.087 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.104+0000 7fe8e38e4640 1 -- 172.21.15.79:0/2479689665 >> v1:172.21.15.79:6790/0 conn(0x7fe8d40349e0 legacy=0x7fe8d40c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.105+0000 7fe8e38e4640 1 -- 172.21.15.79:0/2479689665 shutdown_connections 2024-03-29T10:52:02.088 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.105+0000 7fe8e38e4640 1 -- 172.21.15.79:0/2479689665 >> 172.21.15.79:0/2479689665 conn(0x7fe8d4005910 msgr2=0x7fe8d4005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.090 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.107+0000 7fe8e38e4640 1 -- 172.21.15.79:0/2479689665 shutdown_connections 2024-03-29T10:52:02.090 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.107+0000 7fe8e38e4640 1 -- 172.21.15.79:0/2479689665 wait complete. 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.108+0000 7fe8e38e4640 10 monclient: build_initial_monmap 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.108+0000 7fe8e38e4640 10 monclient: monmap: 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.108+0000 7fe8e38e4640 1 librados: starting msgr at 2024-03-29T10:52:02.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.109+0000 7fe8e38e4640 1 librados: starting objecter 2024-03-29T10:52:02.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.109+0000 7fe8e38e4640 1 Processor -- start 2024-03-29T10:52:02.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.109+0000 7fe8e38e4640 1 -- start start 2024-03-29T10:52:02.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.109+0000 7fe8e38e4640 1 librados: setting wanted keys 2024-03-29T10:52:02.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.109+0000 7fe8e38e4640 1 librados: calling monclient init 2024-03-29T10:52:02.092 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.109+0000 7fe8e38e4640 10 monclient: init 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8e38e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8e38e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8e38e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d402b590 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8e1680640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe8d402b590 0x7fe8d40bfc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:53028/0 (socket says 172.21.15.79:53028) 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8e1680640 1 -- 172.21.15.79:0/4148807800 learned_addr learned my addr 172.21.15.79:0/4148807800 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe8d402b590 2024-03-29T10:52:02.093 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.110+0000 7fe8d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d4034600 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:02.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d40c0680 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:02.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d4158bd0 con 0x7fe8d4034600 2024-03-29T10:52:02.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40bf220 con 0x7fe8d40c0680 2024-03-29T10:52:02.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d40c03a0 con 0x7fe8d402b590 2024-03-29T10:52:02.097 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:02.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.112+0000 7fe8e38e4640 10 monclient(hunting): authenticate will time out at 1241.404519s 2024-03-29T10:52:02.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.114+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951891993 0 0) 0x7fe8d40c03a0 con 0x7fe8d402b590 2024-03-29T10:52:02.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0004eb0 con 0x7fe8d402b590 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe8d4034600 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142396549 0 0) 0x7fe8d4158bd0 con 0x7fe8d4034600 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8e1e81640 1 -- 172.21.15.79:0/4148807800 >> v1:172.21.15.82:6789/0 conn(0x7fe8d40c0680 legacy=0x7fe8d41633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8e1e81640 1 -- 172.21.15.79:0/4148807800 >> v1:172.21.15.82:6789/0 conn(0x7fe8d40c0680 legacy=0x7fe8d41633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8e1e81640 1 --1- 172.21.15.79:0/4148807800 >> v1:172.21.15.82:6789/0 conn(0x7fe8d40c0680 0x7fe8d41633e0 :-1 s=OPENED pgs=499268 cs=1 l=1).handle_message read tag failed 2024-03-29T10:52:02.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.115+0000 7fe8e1e81640 1 --1- 172.21.15.79:0/4148807800 >> v1:172.21.15.82:6789/0 conn(0x7fe8d40c0680 0x7fe8d41633e0 :-1 s=OPENED pgs=499268 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.116+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d40c03a0 con 0x7fe8d4034600 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe8d40c0680 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.82:6789/0 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.117+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368647487 0 0) 0x7fe8c0004eb0 con 0x7fe8d402b590 2024-03-29T10:52:02.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d4158bd0 con 0x7fe8d402b590 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807576454 0 0) 0x7fe8d40c03a0 con 0x7fe8d4034600 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8c0004eb0 con 0x7fe8d4034600 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe8cc004080 con 0x7fe8d402b590 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.106 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe8dc04f400 con 0x7fe8d4034600 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.118+0000 7fe8d1ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:02.107 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4225424224 0 0) 0x7fe8d4158bd0 con 0x7fe8d402b590 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 >> v1:172.21.15.82:6789/0 conn(0x7fe8d40c0680 legacy=0x7fe8d41633e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 >> v1:172.21.15.79:6789/0 conn(0x7fe8d4034600 legacy=0x7fe8d4159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 1 monclient: found mon.c 2024-03-29T10:52:02.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d4159770 con 0x7fe8d402b590 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 5 monclient: authenticate success, global_id 4412 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 10 monclient: _renew_subs 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d40c3530 con 0x7fe8d402b590 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 10 monclient: _renew_subs 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d40c3880 con 0x7fe8d402b590 2024-03-29T10:52:02.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.119+0000 7fe8e38e4640 1 librados: init done 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8cc0041e0 con 0x7fe8d402b590 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe8cc001180 con 0x7fe8d402b590 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d1ffb640 10 monclient: dump: 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:02.110 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.120+0000 7fe8d37fe640 4 set_mon_vals no callback set 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.122+0000 7fe8e38e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.122+0000 7fe8e38e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.122+0000 7fe8e38e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:02.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.122+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc005080 con 0x7fe8d402b590 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.124+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fe8cc0012e0 con 0x7fe8d402b590 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.125+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fe8cc084a40 con 0x7fe8d402b590 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.126+0000 7fe8e0e7f640 10 monclient: get_auth_request con 0x7fe8c006dde0 auth_method 0 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.126+0000 7fe8d1ffb640 10 client.4412.objecter ms_dispatch 0x7fe8dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.126+0000 7fe8d1ffb640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.126+0000 7fe8d1ffb640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.126+0000 7fe8d1ffb640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T10:52:02.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.126+0000 7fe8d1ffb640 10 client.4412.objecter ms_handle_connect 0x7fe8c006dde0 2024-03-29T10:52:02.114 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.133+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8cc050150 con 0x7fe8d402b590 2024-03-29T10:52:02.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.133+0000 7fe8d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:02.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.133+0000 7fe8d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:02.446 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.464+0000 7fe8e38e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:02.446 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.464+0000 7fe8e38e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:02.446 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.464+0000 7fe8e38e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:02.446 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.464+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8bc005330 con 0x7fe8d402b590 2024-03-29T10:52:02.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.465+0000 7fe8d1ffb640 1 -- 172.21.15.79:0/4148807800 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fe8cc052250 con 0x7fe8d402b590 2024-03-29T10:52:02.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.465+0000 7fe8d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:02.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.465+0000 7fe8d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:02.449 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:02.450 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:02.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 10 librados: watch_flush enter 2024-03-29T10:52:02.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 10 librados: watch_flush exit 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 10 client.4412.objecter successfully canceled tick 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 >> v1:172.21.15.79:6816/4290694627 conn(0x7fe8c006dde0 legacy=0x7fe8c00702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 10 monclient: shutdown 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 >> v1:172.21.15.79:6790/0 conn(0x7fe8d402b590 legacy=0x7fe8d40bfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 shutdown_connections 2024-03-29T10:52:02.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.472+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 >> 172.21.15.79:0/4148807800 conn(0x7fe8d4005910 msgr2=0x7fe8d4033d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:02.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.474+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 shutdown_connections 2024-03-29T10:52:02.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.474+0000 7fe8e38e4640 1 -- 172.21.15.79:0/4148807800 wait complete. 2024-03-29T10:52:02.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:02.475+0000 7fe8e38e4640 1 librados: shutdown 2024-03-29T10:52:02.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:09.280 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:09.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.460+0000 7f975da19640 10 monclient: get_monmap_and_config 2024-03-29T10:52:09.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.460+0000 7f975da19640 10 monclient: build_initial_monmap 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.460+0000 7f975da19640 10 monclient: monmap: 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:09.462866+0000 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:09.462866+0000 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:09.444 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.445 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.462+0000 7f975da19640 1 Processor -- start 2024-03-29T10:52:09.445 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.462+0000 7f975da19640 1 -- start start 2024-03-29T10:52:09.445 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.462+0000 7f975da19640 10 monclient: init 2024-03-29T10:52:09.445 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.463+0000 7f975da19640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:09.446 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.463+0000 7f975da19640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:09.446 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.463+0000 7f975da19640 10 monclient(hunting): picked mon.noname-c con 0x7f9750006890 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:09.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.467+0000 7f975da19640 10 monclient(hunting): picked mon.noname-a con 0x7f9750006c70 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:09.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.467+0000 7f975da19640 10 monclient(hunting): picked mon.noname-b con 0x7f975002b4c0 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:09.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.467+0000 7f975da19640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975002c830 con 0x7f9750006c70 2024-03-29T10:52:09.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975d218640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f975002b4c0 0x7f97500cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:40464/0 (socket says 172.21.15.79:40464) 2024-03-29T10:52:09.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975d218640 1 -- 172.21.15.79:0/3691902172 learned_addr learned my addr 172.21.15.79:0/3691902172 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:09.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975002afa0 con 0x7f9750006890 2024-03-29T10:52:09.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975002aba0 con 0x7f975002b4c0 2024-03-29T10:52:09.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975ca17640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f9750006890 0x7f97500c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:48234/0 (socket says 172.21.15.79:48234) 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975da19640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f975da19640 10 monclient(hunting): authenticate will time out at 1248.761427s 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627837098 0 0) 0x7f975002aba0 con 0x7f975002b4c0 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f97577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f97577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f97577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f97577fe640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.468+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9748003520 con 0x7f975002b4c0 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694318965 0 0) 0x7f975002afa0 con 0x7f9750006890 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:52:09.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975002aba0 con 0x7f9750006890 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323975649 0 0) 0x7f9748003520 con 0x7f975002b4c0 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.469+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 >> v1:172.21.15.82:6789/0 conn(0x7f9750006890 legacy=0x7f97500c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 >> v1:172.21.15.79:6789/0 conn(0x7f9750006c70 legacy=0x7f97500c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 1 monclient: found mon.noname-b 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975002ad30 con 0x7f975002b4c0 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:09.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f975804f730 con 0x7f975002b4c0 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f975da19640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f975da19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: dump: 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.457 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f975da19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f975806a3b0 con 0x7f975002b4c0 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 1 -- 172.21.15.79:0/3691902172 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9758075710 con 0x7f975002b4c0 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f97577fe640 10 monclient: dump: 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.458 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f975da19640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.470+0000 7f975da19640 4 set_mon_vals no callback set 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.471+0000 7f975da19640 10 monclient: shutdown 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.471+0000 7f975da19640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.471+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 >> v1:172.21.15.79:6790/0 conn(0x7f975002b4c0 legacy=0x7f97500cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:09.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.471+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 shutdown_connections 2024-03-29T10:52:09.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.471+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 >> 172.21.15.79:0/3691902172 conn(0x7f97500058f0 msgr2=0x7f9750005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:09.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.472+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 shutdown_connections 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.472+0000 7f975da19640 1 -- 172.21.15.79:0/3691902172 wait complete. 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.473+0000 7f975da19640 10 monclient: build_initial_monmap 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.473+0000 7f975da19640 10 monclient: monmap: 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:09.467 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:09.468 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.468 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.473+0000 7f975da19640 1 librados: starting msgr at 2024-03-29T10:52:09.468 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.473+0000 7f975da19640 1 librados: starting objecter 2024-03-29T10:52:09.468 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.474+0000 7f975da19640 1 Processor -- start 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.474+0000 7f975da19640 1 -- start start 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.474+0000 7f975da19640 1 librados: setting wanted keys 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.474+0000 7f975da19640 1 librados: calling monclient init 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.474+0000 7f975da19640 10 monclient: init 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.475+0000 7f975da19640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.475+0000 7f975da19640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.475+0000 7f975da19640 10 monclient(hunting): picked mon.noname-a con 0x7f9750006890 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.475+0000 7f975da19640 10 monclient(hunting): picked mon.noname-c con 0x7f975002b4c0 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.475+0000 7f975da19640 10 monclient(hunting): picked mon.noname-b con 0x7f975015c960 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975da19640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975002fef0 con 0x7f9750006890 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975d218640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f975015c960 0x7f97500bde20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:49912/0 (socket says 172.21.15.79:49912) 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975d218640 1 -- 172.21.15.79:0/2355613095 learned_addr learned my addr 172.21.15.79:0/2355613095 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:09.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 10 client.?.objecter ms_handle_connect 0x7f975015c960 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975002db00 con 0x7f975015c960 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500c3150 con 0x7f975002b4c0 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975da19640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f975da19640 10 monclient(hunting): authenticate will time out at 1248.769447s 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094174729 0 0) 0x7f975002db00 con 0x7f975015c960 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.476+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c004eb0 con 0x7f975015c960 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.477+0000 7f97557fa640 10 client.?.objecter ms_handle_connect 0x7f975002b4c0 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.477+0000 7f97557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:09.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411468582 0 0) 0x7f97500c3150 con 0x7f975002b4c0 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975002db00 con 0x7f975002b4c0 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311386568 0 0) 0x7f975002db00 con 0x7f975002b4c0 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97500c3150 con 0x7f975002b4c0 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f97440018c0 con 0x7f975002b4c0 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): dump: 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.471 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.478+0000 7f97557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953322980 0 0) 0x7f973c004eb0 con 0x7f975015c960 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f975002db00 con 0x7f975015c960 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f975804f400 con 0x7f975015c960 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 10 monclient(hunting): dump: 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:09.472 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1297142073 0 0) 0x7f97500c3150 con 0x7f975002b4c0 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 >> v1:172.21.15.82:6789/0 conn(0x7f975015c960 legacy=0x7f97500bde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.479+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 >> v1:172.21.15.79:6789/0 conn(0x7f9750006890 legacy=0x7f975015c250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 10 monclient: _finish_hunting 0 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 1 monclient: found mon.c 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:09.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500c32e0 con 0x7f975002b4c0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 10 monclient: _finish_auth 0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 10 monclient: _check_auth_tickets 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f97557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f975da19640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f975da19640 10 monclient: _renew_subs 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f975da19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.480+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975002ecf0 con 0x7f975002b4c0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f975da19640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f975da19640 10 monclient: _renew_subs 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f975da19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97500be530 con 0x7f975002b4c0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f975da19640 1 librados: init done 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9744001a20 con 0x7f975002b4c0 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f97557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:09.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f97440038d0 con 0x7f975002b4c0 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f97557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f9756ffd640 4 set_mon_vals no callback set 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f97557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.481+0000 7f97557fa640 10 monclient: dump: 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.483+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f9744003a30 con 0x7f975002b4c0 2024-03-29T10:52:09.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.484+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f9744083850 con 0x7f975002b4c0 2024-03-29T10:52:09.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.484+0000 7f97557fa640 10 client.4418.objecter ms_dispatch 0x7f9758000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:09.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.484+0000 7f97557fa640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:09.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.484+0000 7f97557fa640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:09.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.484+0000 7f97557fa640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T10:52:09.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.486+0000 7f975da19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:09.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.486+0000 7f975da19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:09.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.486+0000 7f975da19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:09.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.486+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9738005080 con 0x7f975002b4c0 2024-03-29T10:52:09.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.486+0000 7f975ca17640 10 monclient: get_auth_request con 0x7f973c0700d0 auth_method 0 2024-03-29T10:52:09.477 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.489+0000 7f97557fa640 10 client.4418.objecter ms_handle_connect 0x7f973c0700d0 2024-03-29T10:52:09.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.499+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f974404ef60 con 0x7f975002b4c0 2024-03-29T10:52:09.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.499+0000 7f97557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:09.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.499+0000 7f97557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:09.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.881+0000 7f975da19640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:09.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.881+0000 7f975da19640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:09.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.881+0000 7f975da19640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:09.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.881+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9738005330 con 0x7f975002b4c0 2024-03-29T10:52:09.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.883+0000 7f97557fa640 1 -- 172.21.15.79:0/2355613095 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f9744051060 con 0x7f975002b4c0 2024-03-29T10:52:09.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.883+0000 7f97557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:09.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.883+0000 7f97557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:09.867 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:09.867 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:09.874 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.888+0000 7f975da19640 10 librados: watch_flush enter 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 10 librados: watch_flush exit 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 10 client.4418.objecter successfully canceled tick 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 >> v1:172.21.15.79:6816/4290694627 conn(0x7f973c0700d0 legacy=0x7f973c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 10 monclient: shutdown 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.889+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 >> v1:172.21.15.79:6790/0 conn(0x7f975002b4c0 legacy=0x7f975002e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.890+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 shutdown_connections 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.890+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 >> 172.21.15.79:0/2355613095 conn(0x7f97500058f0 msgr2=0x7f9750038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.890+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 shutdown_connections 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.890+0000 7f975da19640 1 -- 172.21.15.79:0/2355613095 wait complete. 2024-03-29T10:52:09.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:09.890+0000 7f975da19640 1 librados: shutdown 2024-03-29T10:52:09.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:16.695 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:16.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.861+0000 7fd2972e6640 10 monclient: get_monmap_and_config 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.861+0000 7fd2972e6640 10 monclient: build_initial_monmap 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.861+0000 7fd2972e6640 10 monclient: monmap: 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:16.862071+0000 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:16.862071+0000 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:16.843 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:16.844 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.862+0000 7fd2972e6640 1 Processor -- start 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.862+0000 7fd2972e6640 1 -- start start 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.862+0000 7fd2972e6640 10 monclient: init 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd2972e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd2972e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd2972e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2901457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd2972e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd29013f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd2972e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2900a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:16.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd2972e6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900aaef0 con 0x7fd2900a9d70 2024-03-29T10:52:16.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd295883640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fd2900a9d70 0x7fd29014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:54122/0 (socket says 172.21.15.79:54122) 2024-03-29T10:52:16.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.863+0000 7fd295883640 1 -- 172.21.15.79:0/4139648883 learned_addr learned my addr 172.21.15.79:0/4139648883 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:16.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.864+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd290145bd0 con 0x7fd29013f650 2024-03-29T10:52:16.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.864+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd290145eb0 con 0x7fd2901457f0 2024-03-29T10:52:16.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.864+0000 7fd2972e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:16.846 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.864+0000 7fd2972e6640 10 monclient(hunting): authenticate will time out at 1256.156218s 2024-03-29T10:52:16.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.864+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778689598 0 0) 0x7fd290145bd0 con 0x7fd29013f650 2024-03-29T10:52:16.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:16.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd27c003520 con 0x7fd29013f650 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136342815 0 0) 0x7fd290145eb0 con 0x7fd2901457f0 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd290145bd0 con 0x7fd2901457f0 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498955363 0 0) 0x7fd2900aaef0 con 0x7fd2900a9d70 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:16.856 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd290145eb0 con 0x7fd2900a9d70 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289235594 0 0) 0x7fd27c003520 con 0x7fd29013f650 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 >> v1:172.21.15.79:6790/0 conn(0x7fd2901457f0 legacy=0x7fd29013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 >> v1:172.21.15.79:6789/0 conn(0x7fd2900a9d70 legacy=0x7fd29014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 monclient: found mon.noname-c 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290146340 con 0x7fd29013f650 2024-03-29T10:52:16.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: _finish_auth 0 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd28c001690 con 0x7fd29013f650 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd287fff640 10 monclient: dump: 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:16.858 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd2972e6640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.865+0000 7fd2972e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd28c0017f0 con 0x7fd29013f650 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd287fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd287fff640 1 -- 172.21.15.79:0/4139648883 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd28c001c00 con 0x7fd29013f650 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd287fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd287fff640 10 monclient: dump: 2024-03-29T10:52:16.859 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:16.860 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd2972e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd2972e6640 4 set_mon_vals no callback set 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd2972e6640 10 monclient: shutdown 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd2972e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.866+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 >> v1:172.21.15.82:6789/0 conn(0x7fd29013f650 legacy=0x7fd2900b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.867+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 shutdown_connections 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.867+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 >> 172.21.15.79:0/4139648883 conn(0x7fd290066a00 msgr2=0x7fd290066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.867+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 shutdown_connections 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.867+0000 7fd2972e6640 1 -- 172.21.15.79:0/4139648883 wait complete. 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.867+0000 7fd2972e6640 10 monclient: build_initial_monmap 2024-03-29T10:52:16.861 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.867+0000 7fd2972e6640 10 monclient: monmap: 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 1 librados: starting msgr at 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 1 librados: starting objecter 2024-03-29T10:52:16.862 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 1 Processor -- start 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 1 -- start start 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 1 librados: setting wanted keys 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 1 librados: calling monclient init 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.868+0000 7fd2972e6640 10 monclient: init 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd2972e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd2972e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd2972e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2900a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd2972e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd29013f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd2972e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2901457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd2972e6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29006efa0 con 0x7fd2901457f0 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd295082640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fd2900a9d70 0x7fd2901db7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:42008/0 (socket says 172.21.15.79:42008) 2024-03-29T10:52:16.864 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.869+0000 7fd295082640 1 -- 172.21.15.79:0/1361824326 learned_addr learned my addr 172.21.15.79:0/1361824326 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29006de90 con 0x7fd29013f650 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29006e370 con 0x7fd2900a9d70 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd2972e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd2972e6640 10 monclient(hunting): authenticate will time out at 1256.162172s 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 client.?.objecter ms_handle_connect 0x7fd2900a9d70 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 client.?.objecter ms_handle_connect 0x7fd29013f650 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 client.?.objecter ms_handle_connect 0x7fd2901457f0 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143928674 0 0) 0x7fd29006efa0 con 0x7fd2901457f0 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:16.865 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:16.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:52:16.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd274004eb0 con 0x7fd2901457f0 2024-03-29T10:52:16.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963705308 0 0) 0x7fd29006e370 con 0x7fd2900a9d70 2024-03-29T10:52:16.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:16.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:16.866 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.870+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29006efa0 con 0x7fd2900a9d70 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887525812 0 0) 0x7fd29006de90 con 0x7fd29013f650 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29006e370 con 0x7fd29013f650 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4097758620 0 0) 0x7fd274004eb0 con 0x7fd2901457f0 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd29006de90 con 0x7fd2901457f0 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd2800017f0 con 0x7fd2901457f0 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:16.867 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:16.868 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1409917662 0 0) 0x7fd29006de90 con 0x7fd2901457f0 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 >> v1:172.21.15.79:6790/0 conn(0x7fd2900a9d70 legacy=0x7fd2901db7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 >> v1:172.21.15.82:6789/0 conn(0x7fd29013f650 legacy=0x7fd2901374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 monclient: found mon.a 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29006f210 con 0x7fd2901457f0 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd2972e6640 5 monclient: authenticate success, global_id 4380 2024-03-29T10:52:16.869 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd2972e6640 10 monclient: _renew_subs 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd2972e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd290141e10 con 0x7fd2901457f0 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd280001c40 con 0x7fd2901457f0 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fd280004980 con 0x7fd2901457f0 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd285ffb640 10 monclient: dump: 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:16.870 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd2972e6640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:16.871 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.871+0000 7fd2972e6640 10 monclient: _renew_subs 2024-03-29T10:52:16.872 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.872+0000 7fd2972e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:16.872 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.872+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd290142b30 con 0x7fd2901457f0 2024-03-29T10:52:16.872 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.872+0000 7fd2972e6640 1 librados: init done 2024-03-29T10:52:16.872 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.872+0000 7fd286ffd640 4 set_mon_vals no callback set 2024-03-29T10:52:16.872 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.873+0000 7fd2972e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:16.872 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.873+0000 7fd2972e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:16.873 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.873+0000 7fd2972e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:16.873 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.873+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd290075f10 con 0x7fd2901457f0 2024-03-29T10:52:16.873 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.874+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fd280004ae0 con 0x7fd2901457f0 2024-03-29T10:52:16.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fd280084950 con 0x7fd2901457f0 2024-03-29T10:52:16.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd285ffb640 10 client.4380.objecter ms_dispatch 0x7fd290000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:16.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd285ffb640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:16.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd285ffb640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:16.875 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd285ffb640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T10:52:16.876 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd295082640 10 monclient: get_auth_request con 0x7fd27406df30 auth_method 0 2024-03-29T10:52:16.876 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.875+0000 7fd285ffb640 10 client.4380.objecter ms_handle_connect 0x7fd27406df30 2024-03-29T10:52:16.876 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.883+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd280050230 con 0x7fd2901457f0 2024-03-29T10:52:16.876 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.883+0000 7fd285ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:16.876 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:16.883+0000 7fd285ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:17.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.266+0000 7fd2972e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:17.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.266+0000 7fd2972e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:17.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.266+0000 7fd2972e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:17.248 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.266+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2900b3680 con 0x7fd2901457f0 2024-03-29T10:52:17.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.267+0000 7fd285ffb640 1 -- 172.21.15.79:0/1361824326 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fd280052160 con 0x7fd2901457f0 2024-03-29T10:52:17.249 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.267+0000 7fd285ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:17.249 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:17.249 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:17.250 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.267+0000 7fd285ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:17.253 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.271+0000 7fd2972e6640 10 librados: watch_flush enter 2024-03-29T10:52:17.253 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.271+0000 7fd2972e6640 10 librados: watch_flush exit 2024-03-29T10:52:17.253 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.271+0000 7fd2972e6640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T10:52:17.253 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.272+0000 7fd2972e6640 10 client.4380.objecter successfully canceled tick 2024-03-29T10:52:17.254 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.272+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 >> v1:172.21.15.79:6816/4290694627 conn(0x7fd27406df30 legacy=0x7fd2740703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:17.254 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.272+0000 7fd2972e6640 10 monclient: shutdown 2024-03-29T10:52:17.254 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.272+0000 7fd2972e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:17.254 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.272+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 >> v1:172.21.15.79:6789/0 conn(0x7fd2901457f0 legacy=0x7fd2901e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:17.254 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.272+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 shutdown_connections 2024-03-29T10:52:17.254 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.273+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 >> 172.21.15.79:0/1361824326 conn(0x7fd290066a00 msgr2=0x7fd290066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:17.255 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.273+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 shutdown_connections 2024-03-29T10:52:17.255 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.273+0000 7fd2972e6640 1 -- 172.21.15.79:0/1361824326 wait complete. 2024-03-29T10:52:17.255 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:17.273+0000 7fd2972e6640 1 librados: shutdown 2024-03-29T10:52:17.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:24.077 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.232+0000 7fc40359e640 10 monclient: get_monmap_and_config 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.232+0000 7fc40359e640 10 monclient: build_initial_monmap 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.232+0000 7fc40359e640 10 monclient: monmap: 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:24.233616+0000 2024-03-29T10:52:24.214 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:24.233616+0000 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.215 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.234+0000 7fc40359e640 1 Processor -- start 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.236+0000 7fc40359e640 1 -- start start 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.236+0000 7fc40359e640 10 monclient: init 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3fc006a70 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3fc02b100 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3fc029c30 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc02a270 con 0x7fc3fc006a70 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc02c440 con 0x7fc3fc02b100 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc0c9440 con 0x7fc3fc029c30 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:24.221 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40359e640 10 monclient(hunting): authenticate will time out at 1263.529405s 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40259c640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc3fc006a70 0x7fc3fc0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:56958/0 (socket says 172.21.15.79:56958) 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40259c640 1 -- 172.21.15.79:0/1578164787 learned_addr learned my addr 172.21.15.79:0/1578164787 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc402d9d640 1 --1- 172.21.15.79:0/1578164787 >> v1:172.21.15.79:6790/0 conn(0x7fc3fc029c30 0x7fc3fc0d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:38550/0 (socket says 172.21.15.79:38550) 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943682955 0 0) 0x7fc3fc02a270 con 0x7fc3fc006a70 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40159a640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:52:24.222 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.237+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ec003520 con 0x7fc3fc006a70 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375318706 0 0) 0x7fc3fc0c9440 con 0x7fc3fc029c30 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 10 monclient(hunting): _negotiate 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3fc02a270 con 0x7fc3fc029c30 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3259415253 0 0) 0x7fc3fc02a270 con 0x7fc3fc029c30 2024-03-29T10:52:24.223 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.238+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 >> v1:172.21.15.82:6789/0 conn(0x7fc3fc02b100 legacy=0x7fc3fc0cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:24.224 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.240+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 >> v1:172.21.15.79:6789/0 conn(0x7fc3fc006a70 legacy=0x7fc3fc0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:24.224 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.240+0000 7fc40159a640 10 monclient: _finish_hunting 0 2024-03-29T10:52:24.224 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.240+0000 7fc40159a640 1 monclient: found mon.noname-b 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc0c95d0 con 0x7fc3fc029c30 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: _finish_auth 0 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: _check_auth_tickets 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc40404f730 con 0x7fc3fc029c30 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: dump: 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:24.225 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40359e640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4040008d0 con 0x7fc3fc029c30 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:24.226 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.241+0000 7fc40359e640 4 set_mon_vals no callback set 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40159a640 1 -- 172.21.15.79:0/1578164787 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc4040758b0 con 0x7fc3fc029c30 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40159a640 10 monclient: dump: 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:24.227 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 10 monclient: shutdown 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 1 -- 172.21.15.79:0/1578164787 >> v1:172.21.15.79:6790/0 conn(0x7fc3fc029c30 legacy=0x7fc3fc0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 1 -- 172.21.15.79:0/1578164787 shutdown_connections 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 1 -- 172.21.15.79:0/1578164787 >> 172.21.15.79:0/1578164787 conn(0x7fc3fc0058f0 msgr2=0x7fc3fc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 1 -- 172.21.15.79:0/1578164787 shutdown_connections 2024-03-29T10:52:24.228 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.242+0000 7fc40359e640 1 -- 172.21.15.79:0/1578164787 wait complete. 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.244+0000 7fc40359e640 10 monclient: build_initial_monmap 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.244+0000 7fc40359e640 10 monclient: monmap: 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:24.237 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 1 librados: starting msgr at 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 1 librados: starting objecter 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 1 Processor -- start 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 1 -- start start 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 1 librados: setting wanted keys 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 1 librados: calling monclient init 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.245+0000 7fc40359e640 10 monclient: init 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.246+0000 7fc40359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.246+0000 7fc40359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.246+0000 7fc40359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3fc006a70 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:24.238 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.246+0000 7fc40359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3fc029c30 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc40359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3fc157f30 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc40359e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc0ccbb0 con 0x7fc3fc029c30 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc40359e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc0cdbd0 con 0x7fc3fc006a70 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc40359e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc0cbf80 con 0x7fc3fc157f30 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc40359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc40359e640 10 monclient(hunting): authenticate will time out at 1263.539549s 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc401d9b640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fc3fc029c30 0x7fc3fc0cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:58354/0 (socket says 172.21.15.79:58354) 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc401d9b640 1 -- 172.21.15.79:0/1818446937 learned_addr learned my addr 172.21.15.79:0/1818446937 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc3f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc3fc029c30 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.247+0000 7fc3f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101297584 0 0) 0x7fc3fc0ccbb0 con 0x7fc3fc029c30 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:24.239 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e4004eb0 con 0x7fc3fc029c30 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc3fc157f30 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475868505 0 0) 0x7fc3fc0cbf80 con 0x7fc3fc157f30 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.248+0000 7fc3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3fc0ccbb0 con 0x7fc3fc157f30 2024-03-29T10:52:24.240 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4004175123 0 0) 0x7fc3e4004eb0 con 0x7fc3fc029c30 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3fc0cbf80 con 0x7fc3fc029c30 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc3f40017d0 con 0x7fc3fc029c30 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:24.241 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159635053 0 0) 0x7fc3fc0ccbb0 con 0x7fc3fc157f30 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e4004eb0 con 0x7fc3fc157f30 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc40404f400 con 0x7fc3fc157f30 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:24.242 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc3fc006a70 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744495961 0 0) 0x7fc3e4004eb0 con 0x7fc3fc157f30 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 >> v1:172.21.15.82:6789/0 conn(0x7fc3fc006a70 legacy=0x7fc3fc0cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 >> v1:172.21.15.79:6789/0 conn(0x7fc3fc029c30 legacy=0x7fc3fc0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 monclient: found mon.c 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc0cbc80 con 0x7fc3fc157f30 2024-03-29T10:52:24.243 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc3f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc40359e640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc40359e640 10 monclient: _renew_subs 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc40359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.249+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc16c6b0 con 0x7fc3fc157f30 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.250+0000 7fc40359e640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.250+0000 7fc40359e640 10 monclient: _renew_subs 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.250+0000 7fc40359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.250+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc16ca20 con 0x7fc3fc157f30 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.250+0000 7fc40359e640 1 librados: init done 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc40359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc40359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:24.244 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc40359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc004cd0 con 0x7fc3fc157f30 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc40404fdd0 con 0x7fc3fc157f30 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc3f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc404067bf0 con 0x7fc3fc157f30 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc3f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.251+0000 7fc3f2ffd640 10 monclient: dump: 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:24.245 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.252+0000 7fc400d99640 4 set_mon_vals no callback set 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.254+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fc404067d50 con 0x7fc3fc157f30 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.254+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fc4040ff390 con 0x7fc3fc157f30 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.254+0000 7fc40259c640 10 monclient: get_auth_request con 0x7fc3e406dd30 auth_method 0 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.255+0000 7fc3f2ffd640 10 client.4430.objecter ms_dispatch 0x7fc404000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.255+0000 7fc3f2ffd640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.255+0000 7fc3f2ffd640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.255+0000 7fc3f2ffd640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:52:24.246 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.255+0000 7fc3f2ffd640 10 client.4430.objecter ms_handle_connect 0x7fc3e406dd30 2024-03-29T10:52:24.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.261+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4040cac70 con 0x7fc3fc157f30 2024-03-29T10:52:24.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.261+0000 7fc3f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:24.247 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.261+0000 7fc3f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:24.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.623+0000 7fc40359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:24.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.623+0000 7fc40359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:24.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.623+0000 7fc40359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:24.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.623+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3fc0086f0 con 0x7fc3fc157f30 2024-03-29T10:52:24.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.624+0000 7fc3f2ffd640 1 -- 172.21.15.79:0/1818446937 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fc4040ccba0 con 0x7fc3fc157f30 2024-03-29T10:52:24.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.624+0000 7fc3f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:24.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.624+0000 7fc3f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:24.606 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:24.606 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 10 librados: watch_flush enter 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 10 librados: watch_flush exit 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 >> v1:172.21.15.79:6816/4290694627 conn(0x7fc3e406dd30 legacy=0x7fc3e40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 10 monclient: shutdown 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 >> v1:172.21.15.79:6790/0 conn(0x7fc3fc157f30 legacy=0x7fc3fc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:24.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 shutdown_connections 2024-03-29T10:52:24.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 >> 172.21.15.79:0/1818446937 conn(0x7fc3fc0058f0 msgr2=0x7fc3fc032d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:24.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 shutdown_connections 2024-03-29T10:52:24.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.628+0000 7fc40359e640 1 -- 172.21.15.79:0/1818446937 wait complete. 2024-03-29T10:52:24.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:24.629+0000 7fc40359e640 1 librados: shutdown 2024-03-29T10:52:24.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:31.431 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.601+0000 7ff5f98eb640 10 monclient: get_monmap_and_config 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.601+0000 7ff5f98eb640 10 monclient: build_initial_monmap 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.601+0000 7ff5f98eb640 10 monclient: monmap: 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:31.602215+0000 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:31.602215+0000 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:31.584 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.602+0000 7ff5f98eb640 1 Processor -- start 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.602+0000 7ff5f98eb640 1 -- start start 2024-03-29T10:52:31.585 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.602+0000 7ff5f98eb640 10 monclient: init 2024-03-29T10:52:31.586 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.603+0000 7ff5f98eb640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.603+0000 7ff5f98eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f40acfa0 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f88e9640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7ff5f40acfa0 0x7ff5f4141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:37580/0 (socket says 172.21.15.79:37580) 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f88e9640 1 -- 172.21.15.79:0/3125502704 learned_addr learned my addr 172.21.15.79:0/3125502704 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f40ae220 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f4072f10 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40aa9f0 con 0x7ff5f40ae220 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40ac4c0 con 0x7ff5f4072f10 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40ac7a0 con 0x7ff5f40acfa0 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:31.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.604+0000 7ff5f98eb640 10 monclient(hunting): authenticate will time out at 1270.896930s 2024-03-29T10:52:31.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.607+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820531048 0 0) 0x7ff5f40aa9f0 con 0x7ff5f40ae220 2024-03-29T10:52:31.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.607+0000 7ff5f37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:31.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.607+0000 7ff5f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:31.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.607+0000 7ff5f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:31.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.607+0000 7ff5f37fe640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:52:31.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.607+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5dc004630 con 0x7ff5f40ae220 2024-03-29T10:52:31.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3424355780 0 0) 0x7ff5dc004630 con 0x7ff5f40ae220 2024-03-29T10:52:31.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 >> v1:172.21.15.79:6790/0 conn(0x7ff5f40acfa0 legacy=0x7ff5f4141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 >> v1:172.21.15.82:6789/0 conn(0x7ff5f4072f10 legacy=0x7ff5f40ad380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:31.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 monclient: found mon.noname-a 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40ab5a0 con 0x7ff5f40ae220 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff5e8001690 con 0x7ff5f40ae220 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: dump: 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:31.594 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5e80017f0 con 0x7ff5f40ae220 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 1 -- 172.21.15.79:0/3125502704 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff5e8001c40 con 0x7ff5f40ae220 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f37fe640 10 monclient: dump: 2024-03-29T10:52:31.595 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f98eb640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f98eb640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.608+0000 7ff5f98eb640 4 set_mon_vals no callback set 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 10 monclient: shutdown 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:31.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 >> v1:172.21.15.79:6789/0 conn(0x7ff5f40ae220 legacy=0x7ff5f4141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 shutdown_connections 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 >> 172.21.15.79:0/3125502704 conn(0x7ff5f40b3a30 msgr2=0x7ff5f40b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 shutdown_connections 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.609+0000 7ff5f98eb640 1 -- 172.21.15.79:0/3125502704 wait complete. 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.610+0000 7ff5f98eb640 10 monclient: build_initial_monmap 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.610+0000 7ff5f98eb640 10 monclient: monmap: 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:31.597 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 1 librados: starting msgr at 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 1 librados: starting objecter 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 1 Processor -- start 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 1 -- start start 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 1 librados: setting wanted keys 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 1 librados: calling monclient init 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.611+0000 7ff5f98eb640 10 monclient: init 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f4072f10 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f40acfa0 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:31.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f40ae220 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:31.599 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f412f940 con 0x7ff5f4072f10 2024-03-29T10:52:31.599 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f41f9830 con 0x7ff5f40acfa0 2024-03-29T10:52:31.599 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f412ca60 con 0x7ff5f40ae220 2024-03-29T10:52:31.599 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:31.599 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.612+0000 7ff5f98eb640 10 monclient(hunting): authenticate will time out at 1270.904538s 2024-03-29T10:52:31.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.613+0000 7ff5f90ea640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7ff5f40ae220 0x7ff5f41f9120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:37748/0 (socket says 172.21.15.79:37748) 2024-03-29T10:52:31.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.613+0000 7ff5f90ea640 1 -- 172.21.15.79:0/356819280 learned_addr learned my addr 172.21.15.79:0/356819280 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:31.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 client.?.objecter ms_handle_connect 0x7ff5f40ae220 2024-03-29T10:52:31.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:31.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 client.?.objecter ms_handle_connect 0x7ff5f40acfa0 2024-03-29T10:52:31.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781127385 0 0) 0x7ff5f412ca60 con 0x7ff5f40ae220 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.615+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e0004eb0 con 0x7ff5f40ae220 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284590368 0 0) 0x7ff5f41f9830 con 0x7ff5f40acfa0 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f412ca60 con 0x7ff5f40acfa0 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 89891461 0 0) 0x7ff5f412ca60 con 0x7ff5f40acfa0 2024-03-29T10:52:31.602 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5f41f9830 con 0x7ff5f40acfa0 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff5e8001b00 con 0x7ff5f40acfa0 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): dump: 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.603 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1152947302 0 0) 0x7ff5e0004eb0 con 0x7ff5f40ae220 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5f412ca60 con 0x7ff5f40ae220 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff5e4001c70 con 0x7ff5f40ae220 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient(hunting): dump: 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:31.604 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 951897110 0 0) 0x7ff5f41f9830 con 0x7ff5f40acfa0 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 >> v1:172.21.15.79:6790/0 conn(0x7ff5f40ae220 legacy=0x7ff5f41f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 >> v1:172.21.15.79:6789/0 conn(0x7ff5f4072f10 legacy=0x7ff5f412d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 monclient: found mon.b 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f412cd40 con 0x7ff5f40acfa0 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient: _finish_auth 0 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.616+0000 7ff5f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:31.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f98eb640 5 monclient: authenticate success, global_id 4381 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5e8001d00 con 0x7ff5f40acfa0 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7ff5e8003b60 con 0x7ff5f40acfa0 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f17fa640 10 monclient: dump: 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:31.606 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f2ffd640 4 set_mon_vals no callback set 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f98eb640 10 monclient: _renew_subs 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f98eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.617+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f412d020 con 0x7ff5f40acfa0 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.618+0000 7ff5f98eb640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.618+0000 7ff5f98eb640 10 monclient: _renew_subs 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.618+0000 7ff5f98eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.618+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f413db00 con 0x7ff5f40acfa0 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.618+0000 7ff5f98eb640 1 librados: init done 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.620+0000 7ff5f98eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.620+0000 7ff5f98eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.620+0000 7ff5f98eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.620+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f413e1a0 con 0x7ff5f40acfa0 2024-03-29T10:52:31.607 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.620+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7ff5e8019b30 con 0x7ff5f40acfa0 2024-03-29T10:52:31.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.621+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7ff5e8083880 con 0x7ff5f40acfa0 2024-03-29T10:52:31.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.621+0000 7ff5f17fa640 10 client.4381.objecter ms_dispatch 0x7ff5f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:31.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.621+0000 7ff5f17fa640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:31.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.621+0000 7ff5f17fa640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:31.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.621+0000 7ff5f17fa640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T10:52:31.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.623+0000 7ff5f88e9640 10 monclient: get_auth_request con 0x7ff5e006ded0 auth_method 0 2024-03-29T10:52:31.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.624+0000 7ff5f17fa640 10 client.4381.objecter ms_handle_connect 0x7ff5e006ded0 2024-03-29T10:52:31.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.628+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5e804f160 con 0x7ff5f40acfa0 2024-03-29T10:52:31.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.628+0000 7ff5f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:31.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.628+0000 7ff5f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:31.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.959+0000 7ff5f98eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:31.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.959+0000 7ff5f98eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:31.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.959+0000 7ff5f98eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:52:31.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.959+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5f404f730 con 0x7ff5f40acfa0 2024-03-29T10:52:31.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.960+0000 7ff5f17fa640 1 -- 172.21.15.79:0/356819280 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7ff5e8051090 con 0x7ff5f40acfa0 2024-03-29T10:52:31.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.960+0000 7ff5f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:31.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.960+0000 7ff5f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:31.941 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:31.941 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:31.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.964+0000 7ff5f98eb640 10 librados: watch_flush enter 2024-03-29T10:52:31.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.964+0000 7ff5f98eb640 10 librados: watch_flush exit 2024-03-29T10:52:31.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.964+0000 7ff5f98eb640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T10:52:31.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.964+0000 7ff5f98eb640 10 client.4381.objecter successfully canceled tick 2024-03-29T10:52:31.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 >> v1:172.21.15.79:6816/4290694627 conn(0x7ff5e006ded0 legacy=0x7ff5e0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 10 monclient: shutdown 2024-03-29T10:52:31.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:31.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 >> v1:172.21.15.82:6789/0 conn(0x7ff5f40acfa0 legacy=0x7ff5f41f8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:31.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 shutdown_connections 2024-03-29T10:52:31.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 >> 172.21.15.79:0/356819280 conn(0x7ff5f40b3a30 msgr2=0x7ff5f40b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:31.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 shutdown_connections 2024-03-29T10:52:31.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.966+0000 7ff5f98eb640 1 -- 172.21.15.79:0/356819280 wait complete. 2024-03-29T10:52:31.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:31.967+0000 7ff5f98eb640 1 librados: shutdown 2024-03-29T10:52:31.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:38.773 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:38.926 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.941+0000 7f855f4e0640 10 monclient: get_monmap_and_config 2024-03-29T10:52:38.926 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.941+0000 7f855f4e0640 10 monclient: build_initial_monmap 2024-03-29T10:52:38.926 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.941+0000 7f855f4e0640 10 monclient: monmap: 2024-03-29T10:52:38.926 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:38.942718+0000 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:38.942718+0000 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.943+0000 7f855f4e0640 1 Processor -- start 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.943+0000 7f855f4e0640 1 -- start start 2024-03-29T10:52:38.927 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.943+0000 7f855f4e0640 10 monclient: init 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855f4e0640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855f4e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855f4e0640 10 monclient(hunting): picked mon.noname-c con 0x7f85580a9ac0 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855f4e0640 10 monclient(hunting): picked mon.noname-a con 0x7f855813ee90 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855da7d640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f85580af900 0x7f8558149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43996/0 (socket says 172.21.15.79:43996) 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855da7d640 1 -- 172.21.15.79:0/1261875912 learned_addr learned my addr 172.21.15.79:0/1261875912 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855f4e0640 10 monclient(hunting): picked mon.noname-b con 0x7f85580af900 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:38.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.947+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580afd10 con 0x7f855813ee90 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580af6f0 con 0x7f85580a9ac0 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580aac80 con 0x7f85580af900 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f855f4e0640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f855f4e0640 10 monclient(hunting): authenticate will time out at 1278.240260s 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924158900 0 0) 0x7f85580aac80 con 0x7f85580af900 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8544003440 con 0x7f85580af900 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751941733 0 0) 0x7f8544003440 con 0x7f85580af900 2024-03-29T10:52:38.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.948+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 >> v1:172.21.15.82:6789/0 conn(0x7f85580a9ac0 legacy=0x7f85580adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:38.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 >> v1:172.21.15.79:6789/0 conn(0x7f855813ee90 legacy=0x7f855813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:38.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 1 monclient: found mon.noname-b 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580aaf60 con 0x7f85580af900 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: _finish_auth 0 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8550001a10 con 0x7f85580af900 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: dump: 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:38.933 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f855f4e0640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:52:38.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f855f4e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:38.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8550001be0 con 0x7f85580af900 2024-03-29T10:52:38.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.951+0000 7f854ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:38.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.952+0000 7f854ffff640 1 -- 172.21.15.79:0/1261875912 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8550002030 con 0x7f85580af900 2024-03-29T10:52:38.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.952+0000 7f854ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:38.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.952+0000 7f854ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.952+0000 7f854ffff640 10 monclient: dump: 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.952+0000 7f855f4e0640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.952+0000 7f855f4e0640 4 set_mon_vals no callback set 2024-03-29T10:52:38.936 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.953+0000 7f855f4e0640 10 monclient: shutdown 2024-03-29T10:52:38.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.953+0000 7f855f4e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:38.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.953+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 >> v1:172.21.15.79:6790/0 conn(0x7f85580af900 legacy=0x7f8558149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:38.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.955+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 shutdown_connections 2024-03-29T10:52:38.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.955+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 >> 172.21.15.79:0/1261875912 conn(0x7f85580b35e0 msgr2=0x7f85580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:38.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.955+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 shutdown_connections 2024-03-29T10:52:38.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.955+0000 7f855f4e0640 1 -- 172.21.15.79:0/1261875912 wait complete. 2024-03-29T10:52:38.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.956+0000 7f855f4e0640 10 monclient: build_initial_monmap 2024-03-29T10:52:38.938 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.957+0000 7f855f4e0640 10 monclient: monmap: 2024-03-29T10:52:38.938 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:38.938 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:38.938 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:38.939 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.957+0000 7f855f4e0640 1 librados: starting msgr at 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.957+0000 7f855f4e0640 1 librados: starting objecter 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.957+0000 7f855f4e0640 1 Processor -- start 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.958+0000 7f855f4e0640 1 -- start start 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.958+0000 7f855f4e0640 1 librados: setting wanted keys 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.958+0000 7f855f4e0640 1 librados: calling monclient init 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.958+0000 7f855f4e0640 10 monclient: init 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.959+0000 7f855f4e0640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.959+0000 7f855f4e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.959+0000 7f855f4e0640 10 monclient(hunting): picked mon.noname-a con 0x7f85580a9ac0 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:38.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.959+0000 7f855f4e0640 10 monclient(hunting): picked mon.noname-c con 0x7f85580af900 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:38.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.961+0000 7f855ca7b640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f85580af900 0x7f85581368c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:46856/0 (socket says 172.21.15.79:46856) 2024-03-29T10:52:38.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855f4e0640 10 monclient(hunting): picked mon.noname-b con 0x7f855813ee90 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:38.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855f4e0640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85581d3dd0 con 0x7f85580a9ac0 2024-03-29T10:52:38.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855f4e0640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85581d2ec0 con 0x7f855813ee90 2024-03-29T10:52:38.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855f4e0640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8558140480 con 0x7f85580af900 2024-03-29T10:52:38.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855f4e0640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855f4e0640 10 monclient(hunting): authenticate will time out at 1278.254759s 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855da7d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f855813ee90 0x7f8558140fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:50869/0 (socket says 172.21.15.79:50869) 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.962+0000 7f855da7d640 1 -- 172.21.15.79:0/1696717027 learned_addr learned my addr 172.21.15.79:0/1696717027 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 client.?.objecter ms_handle_connect 0x7f85580af900 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858238497 0 0) 0x7f8558140480 con 0x7f85580af900 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:38.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c004eb0 con 0x7f85580af900 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 client.?.objecter ms_handle_connect 0x7f855813ee90 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.964+0000 7f854dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 client.?.objecter ms_handle_connect 0x7f85580a9ac0 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353224871 0 0) 0x7f85581d3dd0 con 0x7f85580a9ac0 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:52:38.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558140480 con 0x7f85580a9ac0 2024-03-29T10:52:38.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523897447 0 0) 0x7f85581d2ec0 con 0x7f855813ee90 2024-03-29T10:52:38.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:38.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:38.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:38.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85581d3dd0 con 0x7f855813ee90 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3783778770 0 0) 0x7f853c004eb0 con 0x7f85580af900 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85581d2ec0 con 0x7f85580af900 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 304902122 0 0) 0x7f8558140480 con 0x7f85580a9ac0 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853c004eb0 con 0x7f85580a9ac0 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8554001820 con 0x7f85580af900 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.965+0000 7f854dffb640 10 monclient(hunting): dump: 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.951 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f85480016f0 con 0x7f85580a9ac0 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient(hunting): dump: 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:38.952 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2041664765 0 0) 0x7f85581d2ec0 con 0x7f85580af900 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 >> v1:172.21.15.82:6789/0 conn(0x7f855813ee90 legacy=0x7f8558140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:38.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 >> v1:172.21.15.79:6789/0 conn(0x7f85580a9ac0 legacy=0x7f85581d36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 1 monclient: found mon.c 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558140610 con 0x7f85580af900 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:38.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.966+0000 7f854dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:38.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.967+0000 7f855f4e0640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:52:38.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.967+0000 7f855f4e0640 10 monclient: _renew_subs 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.967+0000 7f855f4e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.967+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8558136fd0 con 0x7f85580af900 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.968+0000 7f855f4e0640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.968+0000 7f855f4e0640 10 monclient: _renew_subs 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.968+0000 7f855f4e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.968+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85581418a0 con 0x7f85580af900 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.968+0000 7f855f4e0640 1 librados: init done 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8554001980 con 0x7f85580af900 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8554003790 con 0x7f85580af900 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854dffb640 10 monclient: dump: 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:38.957 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:38.958 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f854effd640 4 set_mon_vals no callback set 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.970+0000 7f855f4e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.971+0000 7f855f4e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.971+0000 7f855f4e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.971+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85581d3050 con 0x7f85580af900 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.972+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f8554003970 con 0x7f85580af900 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.972+0000 7f855d27c640 10 monclient: get_auth_request con 0x7f853c06e000 auth_method 0 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.972+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f8554054640 con 0x7f85580af900 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.972+0000 7f854dffb640 10 client.4442.objecter ms_dispatch 0x7f8558000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.972+0000 7f854dffb640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.972+0000 7f854dffb640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.973+0000 7f854dffb640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T10:52:38.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.973+0000 7f854dffb640 10 client.4442.objecter ms_handle_connect 0x7f853c06e000 2024-03-29T10:52:38.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.982+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85540508e0 con 0x7f85580af900 2024-03-29T10:52:38.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.982+0000 7f854dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:38.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:38.982+0000 7f854dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.311+0000 7f855f4e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.311+0000 7f855f4e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.311+0000 7f855f4e0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.311+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8558137160 con 0x7f85580af900 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.312+0000 7f854dffb640 1 -- 172.21.15.79:0/1696717027 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f85540508e0 con 0x7f85580af900 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.312+0000 7f854dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:39.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.312+0000 7f854dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:39.294 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:39.294 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:39.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.316+0000 7f855f4e0640 10 librados: watch_flush enter 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 10 librados: watch_flush exit 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 10 client.4442.objecter successfully canceled tick 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 >> v1:172.21.15.79:6816/4290694627 conn(0x7f853c06e000 legacy=0x7f853c0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 10 monclient: shutdown 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 >> v1:172.21.15.79:6790/0 conn(0x7f85580af900 legacy=0x7f85581368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 shutdown_connections 2024-03-29T10:52:39.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 >> 172.21.15.79:0/1696717027 conn(0x7f85580b35e0 msgr2=0x7f855813b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:39.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 shutdown_connections 2024-03-29T10:52:39.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.317+0000 7f855f4e0640 1 -- 172.21.15.79:0/1696717027 wait complete. 2024-03-29T10:52:39.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:39.318+0000 7f855f4e0640 1 librados: shutdown 2024-03-29T10:52:39.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:46.124 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:46.269 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.287+0000 7fe1d7fff640 10 monclient: get_monmap_and_config 2024-03-29T10:52:46.269 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.287+0000 7fe1d7fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:46.269 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.287+0000 7fe1d7fff640 10 monclient: monmap: 2024-03-29T10:52:46.269 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:46.269 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:46.288854+0000 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:46.288854+0000 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:46.270 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.271 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.289+0000 7fe1d7fff640 1 Processor -- start 2024-03-29T10:52:46.271 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.289+0000 7fe1d7fff640 1 -- start start 2024-03-29T10:52:46.271 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.289+0000 7fe1d7fff640 10 monclient: init 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.290+0000 7fe1d7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.290+0000 7fe1d7fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d00389e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d0038dc0 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d0030660 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00c54c0 con 0x7fe1d00389e0 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0036430 con 0x7fe1d0038dc0 2024-03-29T10:52:46.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d0039840 con 0x7fe1d0030660 2024-03-29T10:52:46.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:46.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.292+0000 7fe1d7fff640 10 monclient(hunting): authenticate will time out at 1285.584820s 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.293+0000 7fe1d77fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe1d0030660 0x7fe1d00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:37276/0 (socket says 172.21.15.79:37276) 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.293+0000 7fe1d77fe640 1 -- 172.21.15.79:0/1071173814 learned_addr learned my addr 172.21.15.79:0/1071173814 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153197231 0 0) 0x7fe1d0039840 con 0x7fe1d0030660 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:46.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c0003520 con 0x7fe1d0030660 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607969430 0 0) 0x7fe1d00c54c0 con 0x7fe1d00389e0 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.294+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d0039840 con 0x7fe1d00389e0 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410916372 0 0) 0x7fe1d0039840 con 0x7fe1d00389e0 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 >> v1:172.21.15.79:6790/0 conn(0x7fe1d0030660 legacy=0x7fe1d00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 >> v1:172.21.15.82:6789/0 conn(0x7fe1d0038dc0 legacy=0x7fe1d00cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:46.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d0035690 con 0x7fe1d00389e0 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe1c8001540 con 0x7fe1d00389e0 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: dump: 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.282 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d7fff640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1c80017d0 con 0x7fe1d00389e0 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 1 -- 172.21.15.79:0/1071173814 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe1c8001b60 con 0x7fe1d00389e0 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d5ffb640 10 monclient: dump: 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.283 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d7fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.295+0000 7fe1d7fff640 4 set_mon_vals no callback set 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.296+0000 7fe1d7fff640 10 monclient: shutdown 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.296+0000 7fe1d7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:46.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.296+0000 7fe1d7fff640 1 -- 172.21.15.79:0/1071173814 >> v1:172.21.15.79:6789/0 conn(0x7fe1d00389e0 legacy=0x7fe1d0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:46.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.298+0000 7fe1d7fff640 1 -- 172.21.15.79:0/1071173814 shutdown_connections 2024-03-29T10:52:46.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.298+0000 7fe1d7fff640 1 -- 172.21.15.79:0/1071173814 >> 172.21.15.79:0/1071173814 conn(0x7fe1d0005c00 msgr2=0x7fe1d0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:46.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.299+0000 7fe1d7fff640 1 -- 172.21.15.79:0/1071173814 shutdown_connections 2024-03-29T10:52:46.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.299+0000 7fe1d7fff640 1 -- 172.21.15.79:0/1071173814 wait complete. 2024-03-29T10:52:46.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.300+0000 7fe1d7fff640 10 monclient: build_initial_monmap 2024-03-29T10:52:46.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.300+0000 7fe1d7fff640 10 monclient: monmap: 2024-03-29T10:52:46.286 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:46.286 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.300+0000 7fe1d7fff640 1 librados: starting msgr at 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.300+0000 7fe1d7fff640 1 librados: starting objecter 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.301+0000 7fe1d7fff640 1 Processor -- start 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.301+0000 7fe1d7fff640 1 -- start start 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.301+0000 7fe1d7fff640 1 librados: setting wanted keys 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.301+0000 7fe1d7fff640 1 librados: calling monclient init 2024-03-29T10:52:46.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.301+0000 7fe1d7fff640 10 monclient: init 2024-03-29T10:52:46.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.302+0000 7fe1d7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:46.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.302+0000 7fe1d7fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:46.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.305+0000 7fe1d7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d0030660 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:46.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.305+0000 7fe1d6ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe1d0030660 0x7fe1d015ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:34833/0 (socket says 172.21.15.79:34833) 2024-03-29T10:52:46.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.305+0000 7fe1d6ffd640 1 -- 172.21.15.79:0/2788076681 learned_addr learned my addr 172.21.15.79:0/2788076681 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:46.291 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.305+0000 7fe1d7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d00389e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:46.291 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1d7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d002ff60 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:46.291 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00c7ca0 con 0x7fe1d00389e0 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d002d530 con 0x7fe1d0030660 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00c5650 con 0x7fe1d002ff60 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1d7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1d7fff640 10 monclient(hunting): authenticate will time out at 1285.598461s 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 client.?.objecter ms_handle_connect 0x7fe1d0030660 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911791615 0 0) 0x7fe1d002d530 con 0x7fe1d0030660 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4004eb0 con 0x7fe1d0030660 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 client.?.objecter ms_handle_connect 0x7fe1d002ff60 2024-03-29T10:52:46.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935969821 0 0) 0x7fe1d00c5650 con 0x7fe1d002ff60 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d002d530 con 0x7fe1d002ff60 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.306+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120154233 0 0) 0x7fe1c4004eb0 con 0x7fe1d0030660 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d00c5650 con 0x7fe1d0030660 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe1c8001400 con 0x7fe1d0030660 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:46.293 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2667472540 0 0) 0x7fe1d002d530 con 0x7fe1d002ff60 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1c4004eb0 con 0x7fe1d002ff60 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe1d804fdd0 con 0x7fe1d002ff60 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:46.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 598715206 0 0) 0x7fe1c4004eb0 con 0x7fe1d002ff60 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 >> v1:172.21.15.82:6789/0 conn(0x7fe1d0030660 legacy=0x7fe1d015ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.307+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 >> v1:172.21.15.79:6789/0 conn(0x7fe1d00389e0 legacy=0x7fe1d0160660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:46.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 1 monclient: found mon.c 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d00c57e0 con 0x7fe1d002ff60 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 10 monclient: _renew_subs 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d00c83a0 con 0x7fe1d002ff60 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 10 monclient: _renew_subs 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d002d9a0 con 0x7fe1d002ff60 2024-03-29T10:52:46.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.310+0000 7fe1d7fff640 1 librados: init done 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d804f400 con 0x7fe1d002ff60 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1d7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe1d804f730 con 0x7fe1d002ff60 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1bf7fe640 10 monclient: dump: 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:46.299 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1d7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1d7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.311+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d0007b90 con 0x7fe1d002ff60 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.312+0000 7fe1d57fa640 4 set_mon_vals no callback set 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.313+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fe1d8067bf0 con 0x7fe1d002ff60 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.313+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fe1d80ff250 con 0x7fe1d002ff60 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.313+0000 7fe1bf7fe640 10 client.4448.objecter ms_dispatch 0x7fe1d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.313+0000 7fe1d6ffd640 10 monclient: get_auth_request con 0x7fe1c406dc80 auth_method 0 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.313+0000 7fe1bf7fe640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.313+0000 7fe1bf7fe640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:46.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.314+0000 7fe1bf7fe640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T10:52:46.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.314+0000 7fe1bf7fe640 10 client.4448.objecter ms_handle_connect 0x7fe1c406dc80 2024-03-29T10:52:46.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.320+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1d80cab30 con 0x7fe1d002ff60 2024-03-29T10:52:46.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.320+0000 7fe1bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:46.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.321+0000 7fe1bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.649+0000 7fe1d7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.649+0000 7fe1d7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.649+0000 7fe1d7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.649+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1d0005120 con 0x7fe1d002ff60 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.651+0000 7fe1bf7fe640 1 -- 172.21.15.79:0/2788076681 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fe1d80cca60 con 0x7fe1d002ff60 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.651+0000 7fe1bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:46.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.651+0000 7fe1bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:46.634 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:46.634 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:46.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 10 librados: watch_flush enter 2024-03-29T10:52:46.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 10 librados: watch_flush exit 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 10 client.4448.objecter successfully canceled tick 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 >> v1:172.21.15.79:6816/4290694627 conn(0x7fe1c406dc80 legacy=0x7fe1c4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 10 monclient: shutdown 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 >> v1:172.21.15.79:6790/0 conn(0x7fe1d002ff60 legacy=0x7fe1d01672d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 shutdown_connections 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.655+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 >> 172.21.15.79:0/2788076681 conn(0x7fe1d0005c00 msgr2=0x7fe1d00c4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:46.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.656+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 shutdown_connections 2024-03-29T10:52:46.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.656+0000 7fe1d7fff640 1 -- 172.21.15.79:0/2788076681 wait complete. 2024-03-29T10:52:46.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:46.656+0000 7fe1d7fff640 1 librados: shutdown 2024-03-29T10:52:46.655 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:53.463 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:53.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.627+0000 7f58b7458640 10 monclient: get_monmap_and_config 2024-03-29T10:52:53.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.627+0000 7f58b7458640 10 monclient: build_initial_monmap 2024-03-29T10:52:53.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.627+0000 7f58b7458640 10 monclient: monmap: 2024-03-29T10:52:53.611 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:53.611 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:53.611 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:52:53.629741+0000 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:52:53.629741+0000 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:52:53.612 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.630+0000 7f58b7458640 1 Processor -- start 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.630+0000 7f58b7458640 1 -- start start 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.630+0000 7f58b7458640 10 monclient: init 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient(hunting): picked mon.noname-c con 0x7f58a8034850 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient(hunting): picked mon.noname-a con 0x7f58a802a640 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient(hunting): picked mon.noname-b con 0x7f58a80327d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a802c820 con 0x7f58a802a640 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a802a120 con 0x7f58a8034850 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a80c75b0 con 0x7f58a80327d0 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b7458640 10 monclient(hunting): authenticate will time out at 1292.925404s 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b6c57640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f58a80327d0 0x7f58a80cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:58354/0 (socket says 172.21.15.79:58354) 2024-03-29T10:52:53.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.632+0000 7f58b6c57640 1 -- 172.21.15.79:0/1893690298 learned_addr learned my addr 172.21.15.79:0/1893690298 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:53.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481973694 0 0) 0x7f58a80c75b0 con 0x7f58a80327d0 2024-03-29T10:52:53.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient(hunting): _negotiate 2024-03-29T10:52:53.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:53.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:53.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:52:53.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f589c003520 con 0x7f58a80327d0 2024-03-29T10:52:53.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422653875 0 0) 0x7f589c003520 con 0x7f58a80327d0 2024-03-29T10:52:53.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 >> v1:172.21.15.82:6789/0 conn(0x7f58a8034850 legacy=0x7f58a80c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 >> v1:172.21.15.79:6789/0 conn(0x7f58a802a640 legacy=0x7f58a80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: _finish_hunting 0 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 monclient: found mon.noname-b 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a8033110 con 0x7f58a80327d0 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: _finish_auth 0 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: _check_auth_tickets 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:53.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b7458640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b7458640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f58a0001690 con 0x7f58a80327d0 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.636+0000 7f58b5454640 10 monclient: dump: 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:53.621 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:53.622 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:53.622 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.622 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.637+0000 7f58b7458640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:53.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.639+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58a0001860 con 0x7f58a80327d0 2024-03-29T10:52:53.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.639+0000 7f58b5454640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:53.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.639+0000 7f58b5454640 1 -- 172.21.15.79:0/1893690298 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f58a0001cb0 con 0x7f58a80327d0 2024-03-29T10:52:53.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.639+0000 7f58b5454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.639+0000 7f58b5454640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.639+0000 7f58b5454640 10 monclient: dump: 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:53.624 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:53.625 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.640+0000 7f58b7458640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:53.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.640+0000 7f58b7458640 4 set_mon_vals no callback set 2024-03-29T10:52:53.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.640+0000 7f58b7458640 10 monclient: shutdown 2024-03-29T10:52:53.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.640+0000 7f58b7458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:53.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.640+0000 7f58b7458640 1 -- 172.21.15.79:0/1893690298 >> v1:172.21.15.79:6790/0 conn(0x7f58a80327d0 legacy=0x7f58a80cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.644+0000 7f58b7458640 1 -- 172.21.15.79:0/1893690298 shutdown_connections 2024-03-29T10:52:53.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.644+0000 7f58b7458640 1 -- 172.21.15.79:0/1893690298 >> 172.21.15.79:0/1893690298 conn(0x7f58a80058f0 msgr2=0x7f58a8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:53.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.644+0000 7f58b7458640 1 -- 172.21.15.79:0/1893690298 shutdown_connections 2024-03-29T10:52:53.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.644+0000 7f58b7458640 1 -- 172.21.15.79:0/1893690298 wait complete. 2024-03-29T10:52:53.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.645+0000 7f58b7458640 10 monclient: build_initial_monmap 2024-03-29T10:52:53.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.645+0000 7f58b7458640 10 monclient: monmap: 2024-03-29T10:52:53.627 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:52:53.627 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:53.627 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:52:53.628 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.645+0000 7f58b7458640 1 librados: starting msgr at 2024-03-29T10:52:53.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.645+0000 7f58b7458640 1 librados: starting objecter 2024-03-29T10:52:53.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 1 Processor -- start 2024-03-29T10:52:53.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 1 -- start start 2024-03-29T10:52:53.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 1 librados: setting wanted keys 2024-03-29T10:52:53.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 1 librados: calling monclient init 2024-03-29T10:52:53.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 10 monclient: init 2024-03-29T10:52:53.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:53.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:53.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.646+0000 7f58b7458640 10 monclient(hunting): picked mon.noname-c con 0x7f58a802a640 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:53.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.649+0000 7f58b7458640 10 monclient(hunting): picked mon.noname-a con 0x7f58a80327d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:53.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.649+0000 7f58b5c55640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f58a80327d0 0x7f58a80bf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:39681/0 (socket says 172.21.15.79:39681) 2024-03-29T10:52:53.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b7458640 10 monclient(hunting): picked mon.noname-b con 0x7f58a8034850 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:53.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b7458640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a80c1b50 con 0x7f58a80327d0 2024-03-29T10:52:53.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b7458640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a80befb0 con 0x7f58a8034850 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b7458640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a80c2610 con 0x7f58a802a640 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b7458640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b7458640 10 monclient(hunting): authenticate will time out at 1292.946045s 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.652+0000 7f58b5c55640 1 -- 172.21.15.79:0/910143629 learned_addr learned my addr 172.21.15.79:0/910143629 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 client.?.objecter ms_handle_connect 0x7f58a80327d0 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 client.?.objecter ms_handle_connect 0x7f58a802a640 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956042445 0 0) 0x7f58a80c1b50 con 0x7f58a80327d0 2024-03-29T10:52:53.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.653+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5894004eb0 con 0x7f58a80327d0 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835983660 0 0) 0x7f58a80c2610 con 0x7f58a802a640 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a80c1b50 con 0x7f58a802a640 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2351316059 0 0) 0x7f5894004eb0 con 0x7f58a80327d0 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58a80c2610 con 0x7f58a80327d0 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f58b004fdd0 con 0x7f58a80327d0 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:53.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.654+0000 7f58a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 909389640 0 0) 0x7f58a80c1b50 con 0x7f58a802a640 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5894004eb0 con 0x7f58a802a640 2024-03-29T10:52:53.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f58ac0017d0 con 0x7f58a802a640 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient(hunting): dump: 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2477974657 0 0) 0x7f58a80c2610 con 0x7f58a80327d0 2024-03-29T10:52:53.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6790/0 conn(0x7f58a802a640 legacy=0x7f58a80c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.82:6789/0 conn(0x7f58a8034850 legacy=0x7f58a816b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 monclient: found mon.a 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80c16d0 con 0x7f58a80327d0 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 5 monclient: authenticate success, global_id 4401 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 10 monclient: _renew_subs 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:53.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a80bfea0 con 0x7f58a80327d0 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 10 monclient: _renew_subs 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a814f530 con 0x7f58a80327d0 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b7458640 1 librados: init done 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b5c55640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6789/0 conn(0x7f58a80327d0 legacy=0x7f58a80bf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b5c55640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6789/0 conn(0x7f58a80327d0 legacy=0x7f58a80bf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b5c55640 1 --1- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6789/0 conn(0x7f58a80327d0 0x7f58a80bf790 :-1 s=OPENED pgs=480466 cs=1 l=1).handle_message read tag failed 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58b5c55640 1 --1- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6789/0 conn(0x7f58a80327d0 0x7f58a80bf790 :-1 s=OPENED pgs=480466 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.79:6789/0 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6789/0 conn(0x7f58a80327d0 legacy=0x7f58a80bf790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.655+0000 7f58a6ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:53.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 10 monclient(hunting): picked mon.b con 0x7f58a8034850 addr v1:172.21.15.82:6789/0 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 10 monclient(hunting): picked mon.c con 0x7f58a802a640 addr v1:172.21.15.79:6790/0 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 10 monclient(hunting): picked mon.a con 0x7f589400dae0 addr v1:172.21.15.79:6789/0 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f58a80c2610 con 0x7f589400dae0 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f58a80befb0 con 0x7f58a8034850 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5894004eb0 con 0x7f58a802a640 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 10 client.4401.objecter ms_handle_connect 0x7f58a802a640 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.656+0000 7f58a6ffd640 10 client.4401.objecter resend_mon_ops 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.657+0000 7f58b7458640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:53.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.657+0000 7f58b7458640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:53.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.659+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648138944 0 0) 0x7f5894004eb0 con 0x7f58a802a640 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.659+0000 7f58a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.659+0000 7f58a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.659+0000 7f58a6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5894013b30 con 0x7f58a802a640 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3496096786 0 0) 0x7f5894013b30 con 0x7f58a802a640 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.82:6789/0 conn(0x7f58a8034850 legacy=0x7f58940063f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6789/0 conn(0x7f589400dae0 legacy=0x7f589400ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 1 monclient: found mon.c 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.660+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f58a80c1b50 con 0x7f58a802a640 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:53.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a8004cd0 con 0x7f58a802a640 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a80c2610 con 0x7f58a802a640 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f58b0075710 con 0x7f58a802a640 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.661+0000 7f58a6ffd640 10 monclient: dump: 2024-03-29T10:52:53.653 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.664+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b004f400 con 0x7f58a802a640 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.665+0000 7f58a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.665+0000 7f58b4c53640 4 set_mon_vals no callback set 2024-03-29T10:52:53.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.665+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f58b0067ec0 con 0x7f58a802a640 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.666+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f58b00ff860 con 0x7f58a802a640 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.666+0000 7f58a6ffd640 10 client.4401.objecter ms_dispatch 0x7f58b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.666+0000 7f58a6ffd640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.666+0000 7f58a6ffd640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.667+0000 7f58a6ffd640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.667+0000 7f58b6456640 10 monclient: get_auth_request con 0x7f58a80327d0 auth_method 0 2024-03-29T10:52:53.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.667+0000 7f58a6ffd640 10 client.4401.objecter ms_handle_connect 0x7f58a80327d0 2024-03-29T10:52:53.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.672+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58b00cb140 con 0x7f58a802a640 2024-03-29T10:52:53.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.672+0000 7f58a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:53.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.672+0000 7f58a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:53.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.680+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58b00cd070 con 0x7f58a802a640 2024-03-29T10:52:53.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:53.680+0000 7f58a6ffd640 10 monclient: handle_mon_command_ack 1 not found 2024-03-29T10:52:53.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.004+0000 7f58b7458640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:53.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.004+0000 7f58b7458640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:53.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.004+0000 7f58b7458640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:52:53.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.004+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58a814f1a0 con 0x7f58a802a640 2024-03-29T10:52:53.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.007+0000 7f58a6ffd640 1 -- 172.21.15.79:0/910143629 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f58b00ce890 con 0x7f58a802a640 2024-03-29T10:52:53.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.007+0000 7f58a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:53.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.007+0000 7f58a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:53.989 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:52:53.990 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:53.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 10 librados: watch_flush enter 2024-03-29T10:52:53.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 10 librados: watch_flush exit 2024-03-29T10:52:53.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 10 client.4401.objecter successfully canceled tick 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6816/4290694627 conn(0x7f58a80327d0 legacy=0x7f589407b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 10 monclient: shutdown 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.010+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 >> v1:172.21.15.79:6790/0 conn(0x7f58a802a640 legacy=0x7f589400c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.011+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 shutdown_connections 2024-03-29T10:52:53.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.011+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 >> 172.21.15.79:0/910143629 conn(0x7f58a80058f0 msgr2=0x7f58a80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:53.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.012+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 shutdown_connections 2024-03-29T10:52:53.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.012+0000 7f58b7458640 1 -- 172.21.15.79:0/910143629 wait complete. 2024-03-29T10:52:53.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:52:54.012+0000 7f58b7458640 1 librados: shutdown 2024-03-29T10:52:54.011 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:00.819 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:00.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.985+0000 7f3820892640 10 monclient: get_monmap_and_config 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.985+0000 7f3820892640 10 monclient: build_initial_monmap 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.985+0000 7f3820892640 10 monclient: monmap: 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:00.986061+0000 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:00.986061+0000 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:00.967 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:00.968 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:00.968 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:00.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.985+0000 7f3820892640 1 Processor -- start 2024-03-29T10:53:00.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.986+0000 7f3820892640 1 -- start start 2024-03-29T10:53:00.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.986+0000 7f3820892640 10 monclient: init 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.986+0000 7f3820892640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.986+0000 7f3820892640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.986+0000 7f3820892640 10 monclient(hunting): picked mon.noname-b con 0x7f381c06ade0 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.986+0000 7f3820892640 10 monclient(hunting): picked mon.noname-a con 0x7f381c0af6c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.987+0000 7f3820892640 10 monclient(hunting): picked mon.noname-c con 0x7f381c13f1e0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.987+0000 7f3820892640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c072690 con 0x7f381c0af6c0 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.987+0000 7f3820892640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c0b0f50 con 0x7f381c13f1e0 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.987+0000 7f3820892640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c0afac0 con 0x7f381c06ade0 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.987+0000 7f3820892640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:00.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.987+0000 7f3820892640 10 monclient(hunting): authenticate will time out at 1300.279150s 2024-03-29T10:53:00.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.990+0000 7f381b59e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f381c13f1e0 0x7f381c13f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:55365/0 (socket says 172.21.15.79:55365) 2024-03-29T10:53:00.971 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.990+0000 7f381b59e640 1 -- 172.21.15.79:0/3365165539 learned_addr learned my addr 172.21.15.79:0/3365165539 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025638634 0 0) 0x7f381c0b0f50 con 0x7f381c13f1e0 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804003520 con 0x7f381c13f1e0 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933392040 0 0) 0x7f381c0afac0 con 0x7f381c06ade0 2024-03-29T10:53:00.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.994+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381c0b0f50 con 0x7f381c06ade0 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.995+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2978965870 0 0) 0x7f3804003520 con 0x7f381c13f1e0 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.995+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 >> v1:172.21.15.79:6790/0 conn(0x7f381c06ade0 legacy=0x7f381c1420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:00.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.995+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 >> v1:172.21.15.79:6789/0 conn(0x7f381c0af6c0 legacy=0x7f381c071f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:00.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:53:00.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 1 monclient: found mon.noname-c 2024-03-29T10:53:00.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c151a00 con 0x7f381c13f1e0 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 10 monclient: _finish_auth 0 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3820892640 5 monclient: authenticate success, global_id 4390 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3820892640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3814001a10 con 0x7f381c13f1e0 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.996+0000 7f3819d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3819d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3819d9b640 10 monclient: dump: 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:00.980 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3820892640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3814001c30 con 0x7f381c13f1e0 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3819d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3820892640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.997+0000 7f3820892640 4 set_mon_vals no callback set 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.998+0000 7f3820892640 10 monclient: shutdown 2024-03-29T10:53:00.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.998+0000 7f3820892640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:00.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.998+0000 7f3819d9b640 1 -- 172.21.15.79:0/3365165539 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3814001f40 con 0x7f381c13f1e0 2024-03-29T10:53:00.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.998+0000 7f3820892640 1 -- 172.21.15.79:0/3365165539 >> v1:172.21.15.82:6789/0 conn(0x7f381c13f1e0 legacy=0x7f381c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:00.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.998+0000 7f3819d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:00.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.999+0000 7f3820892640 1 -- 172.21.15.79:0/3365165539 shutdown_connections 2024-03-29T10:53:00.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.999+0000 7f3820892640 1 -- 172.21.15.79:0/3365165539 >> 172.21.15.79:0/3365165539 conn(0x7f381c067420 msgr2=0x7f381c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:00.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.999+0000 7f3820892640 1 -- 172.21.15.79:0/3365165539 shutdown_connections 2024-03-29T10:53:00.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:00.999+0000 7f3820892640 1 -- 172.21.15.79:0/3365165539 wait complete. 2024-03-29T10:53:00.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.001+0000 7f3820892640 10 monclient: build_initial_monmap 2024-03-29T10:53:00.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.001+0000 7f3820892640 10 monclient: monmap: 2024-03-29T10:53:00.983 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:00.983 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.001+0000 7f3820892640 1 librados: starting msgr at 2024-03-29T10:53:00.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.001+0000 7f3820892640 1 librados: starting objecter 2024-03-29T10:53:00.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.001+0000 7f3820892640 1 Processor -- start 2024-03-29T10:53:00.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 1 -- start start 2024-03-29T10:53:00.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 1 librados: setting wanted keys 2024-03-29T10:53:00.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 1 librados: calling monclient init 2024-03-29T10:53:00.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient: init 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient(hunting): picked mon.noname-b con 0x7f381c06ade0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient(hunting): picked mon.noname-a con 0x7f381c0af6c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient(hunting): picked mon.noname-c con 0x7f381c13f1e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c06fbf0 con 0x7f381c0af6c0 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c06eb60 con 0x7f381c06ade0 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c070110 con 0x7f381c13f1e0 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:00.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.002+0000 7f3820892640 10 monclient(hunting): authenticate will time out at 1300.294983s 2024-03-29T10:53:00.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.007+0000 7f381ad9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f381c06ade0 0x7f381c072160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:55443/0 (socket says 172.21.15.79:55443) 2024-03-29T10:53:00.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.007+0000 7f381ad9d640 1 -- 172.21.15.79:0/3869949622 learned_addr learned my addr 172.21.15.79:0/3869949622 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.007+0000 7f38037fe640 10 client.?.objecter ms_handle_connect 0x7f381c06ade0 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.007+0000 7f38037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.007+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821866258 0 0) 0x7f381c06eb60 con 0x7f381c06ade0 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:53:00.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3808004eb0 con 0x7f381c06ade0 2024-03-29T10:53:00.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 10 client.?.objecter ms_handle_connect 0x7f381c13f1e0 2024-03-29T10:53:00.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.008+0000 7f38037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:00.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357538777 0 0) 0x7f3808004eb0 con 0x7f381c06ade0 2024-03-29T10:53:00.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f381c06eb60 con 0x7f381c06ade0 2024-03-29T10:53:00.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500154765 0 0) 0x7f381c070110 con 0x7f381c13f1e0 2024-03-29T10:53:00.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3808004eb0 con 0x7f381c13f1e0 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3810001a50 con 0x7f381c06ade0 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): dump: 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:00.994 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:00.995 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:00.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.012+0000 7f38037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:00.999 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.013+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3955133478 0 0) 0x7f381c06eb60 con 0x7f381c06ade0 2024-03-29T10:53:00.999 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.013+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 >> v1:172.21.15.79:6790/0 conn(0x7f381c13f1e0 legacy=0x7f381c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.013+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 >> v1:172.21.15.79:6789/0 conn(0x7f381c0af6c0 legacy=0x7f381c1e3ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 1 monclient: found mon.b 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c1e41f0 con 0x7f381c06ade0 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f38037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 5 monclient: authenticate success, global_id 4393 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 10 monclient: _renew_subs 2024-03-29T10:53:01.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c1e44a0 con 0x7f381c06ade0 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 10 monclient: _renew_subs 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c13c450 con 0x7f381c06ade0 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.014+0000 7f3820892640 1 librados: init done 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3810001cf0 con 0x7f381c06ade0 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f38037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3810003a40 con 0x7f381c06ade0 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f38037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:01.001 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f38037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f38037fe640 10 monclient: dump: 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:01.002 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.015+0000 7f381959a640 4 set_mon_vals no callback set 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.018+0000 7f3820892640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.018+0000 7f3820892640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.018+0000 7f3820892640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.018+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc005080 con 0x7f381c06ade0 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.019+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f3810003ba0 con 0x7f381c06ade0 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.019+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f3810084b30 con 0x7f381c06ade0 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.019+0000 7f38037fe640 10 client.4393.objecter ms_dispatch 0x7f381c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.020+0000 7f381a59c640 10 monclient: get_auth_request con 0x7f380806db20 auth_method 0 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.021+0000 7f38037fe640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:01.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.021+0000 7f38037fe640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:01.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.021+0000 7f38037fe640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T10:53:01.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.021+0000 7f38037fe640 10 client.4393.objecter ms_handle_connect 0x7f380806db20 2024-03-29T10:53:01.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.027+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3810050410 con 0x7f381c06ade0 2024-03-29T10:53:01.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.027+0000 7f38037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:01.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.027+0000 7f38037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:01.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.351+0000 7f3820892640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:01.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.351+0000 7f3820892640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:01.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.351+0000 7f3820892640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:01.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.351+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37fc005330 con 0x7f381c06ade0 2024-03-29T10:53:01.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.353+0000 7f38037fe640 1 -- 172.21.15.79:0/3869949622 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f3810052340 con 0x7f381c06ade0 2024-03-29T10:53:01.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.353+0000 7f38037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:01.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.353+0000 7f38037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:01.335 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:01.335 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:01.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.357+0000 7f3820892640 10 librados: watch_flush enter 2024-03-29T10:53:01.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.357+0000 7f3820892640 10 librados: watch_flush exit 2024-03-29T10:53:01.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.357+0000 7f3820892640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T10:53:01.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.357+0000 7f3820892640 10 client.4393.objecter successfully canceled tick 2024-03-29T10:53:01.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.357+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 >> v1:172.21.15.79:6816/4290694627 conn(0x7f380806db20 legacy=0x7f380806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 10 monclient: shutdown 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 >> v1:172.21.15.82:6789/0 conn(0x7f381c06ade0 legacy=0x7f381c072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 shutdown_connections 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 >> 172.21.15.79:0/3869949622 conn(0x7f381c067420 msgr2=0x7f381c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 shutdown_connections 2024-03-29T10:53:01.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.358+0000 7f3820892640 1 -- 172.21.15.79:0/3869949622 wait complete. 2024-03-29T10:53:01.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:01.360+0000 7f3820892640 1 librados: shutdown 2024-03-29T10:53:01.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:08.167 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:08.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.327+0000 7faa87df4640 10 monclient: get_monmap_and_config 2024-03-29T10:53:08.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.327+0000 7faa87df4640 10 monclient: build_initial_monmap 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.327+0000 7faa87df4640 10 monclient: monmap: 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:08.328304+0000 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:08.328304+0000 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:08.309 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:08.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.329+0000 7faa87df4640 1 Processor -- start 2024-03-29T10:53:08.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.329+0000 7faa87df4640 1 -- start start 2024-03-29T10:53:08.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.329+0000 7faa87df4640 10 monclient: init 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient(hunting): picked mon.noname-c con 0x7faa801422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient(hunting): picked mon.noname-b con 0x7faa80142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient(hunting): picked mon.noname-a con 0x7faa800b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa80142c80 con 0x7faa800b17d0 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa800aae40 con 0x7faa801422b0 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8013fbd0 con 0x7faa80142690 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:08.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.330+0000 7faa87df4640 10 monclient(hunting): authenticate will time out at 1307.622757s 2024-03-29T10:53:08.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa8538f640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7faa80142690 0x7faa800b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47628/0 (socket says 172.21.15.79:47628) 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa8538f640 1 -- 172.21.15.79:0/2828791223 learned_addr learned my addr 172.21.15.79:0/2828791223 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247123955 0 0) 0x7faa8013fbd0 con 0x7faa80142690 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 10 monclient(hunting): _negotiate 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c003520 con 0x7faa80142690 2024-03-29T10:53:08.316 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 884290437 0 0) 0x7faa6c003520 con 0x7faa80142690 2024-03-29T10:53:08.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.332+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 >> v1:172.21.15.82:6789/0 conn(0x7faa801422b0 legacy=0x7faa8013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:08.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.334+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 >> v1:172.21.15.79:6789/0 conn(0x7faa800b17d0 legacy=0x7faa800b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: _finish_hunting 0 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 1 monclient: found mon.noname-b 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa80151940 con 0x7faa80142690 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: _finish_auth 0 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: _check_auth_tickets 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faa70001690 con 0x7faa80142690 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa87df4640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa87df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:08.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: dump: 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa87df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa700018f0 con 0x7faa80142690 2024-03-29T10:53:08.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:08.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa84b8e640 1 -- 172.21.15.79:0/2828791223 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faa70001cc0 con 0x7faa80142690 2024-03-29T10:53:08.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa87df4640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:08.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.339+0000 7faa87df4640 4 set_mon_vals no callback set 2024-03-29T10:53:08.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.340+0000 7faa84b8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.340+0000 7faa84b8e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.340+0000 7faa84b8e640 10 monclient: dump: 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.340+0000 7faa87df4640 10 monclient: shutdown 2024-03-29T10:53:08.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.340+0000 7faa87df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.340+0000 7faa87df4640 1 -- 172.21.15.79:0/2828791223 >> v1:172.21.15.79:6790/0 conn(0x7faa80142690 legacy=0x7faa800b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.341+0000 7faa87df4640 1 -- 172.21.15.79:0/2828791223 shutdown_connections 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.341+0000 7faa87df4640 1 -- 172.21.15.79:0/2828791223 >> 172.21.15.79:0/2828791223 conn(0x7faa80066a00 msgr2=0x7faa80066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.341+0000 7faa87df4640 1 -- 172.21.15.79:0/2828791223 shutdown_connections 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.341+0000 7faa87df4640 1 -- 172.21.15.79:0/2828791223 wait complete. 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.342+0000 7faa87df4640 10 monclient: build_initial_monmap 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.342+0000 7faa87df4640 10 monclient: monmap: 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:08.325 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:08.326 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:08.326 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 1 librados: starting msgr at 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 1 librados: starting objecter 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 1 Processor -- start 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 1 -- start start 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 1 librados: setting wanted keys 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 1 librados: calling monclient init 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.343+0000 7faa87df4640 10 monclient: init 2024-03-29T10:53:08.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient(hunting): picked mon.noname-a con 0x7faa800b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient(hunting): picked mon.noname-c con 0x7faa801422b0 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient(hunting): picked mon.noname-b con 0x7faa80142690 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa800ab800 con 0x7faa800b17d0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8014e4b0 con 0x7faa80142690 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa800acdd0 con 0x7faa801422b0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.344+0000 7faa87df4640 10 monclient(hunting): authenticate will time out at 1307.636354s 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.345+0000 7faa8538f640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7faa801422b0 0x7faa8014d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:48328/0 (socket says 172.21.15.79:48328) 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.345+0000 7faa8538f640 1 -- 172.21.15.79:0/307771462 learned_addr learned my addr 172.21.15.79:0/307771462 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.345+0000 7faa767fc640 10 client.?.objecter ms_handle_connect 0x7faa801422b0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.345+0000 7faa767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320785821 0 0) 0x7faa800acdd0 con 0x7faa801422b0 2024-03-29T10:53:08.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa64004eb0 con 0x7faa801422b0 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 10 client.?.objecter ms_handle_connect 0x7faa80142690 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.347+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215188058 0 0) 0x7faa64004eb0 con 0x7faa801422b0 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa800acdd0 con 0x7faa801422b0 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faa70001cc0 con 0x7faa801422b0 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): dump: 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:08.332 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812344272 0 0) 0x7faa8014e4b0 con 0x7faa80142690 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.348+0000 7faa767fc640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.349+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa64004eb0 con 0x7faa80142690 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.349+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1558530822 0 0) 0x7faa800acdd0 con 0x7faa801422b0 2024-03-29T10:53:08.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.349+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 >> v1:172.21.15.82:6789/0 conn(0x7faa80142690 legacy=0x7faa8014dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:08.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.349+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 >> v1:172.21.15.79:6789/0 conn(0x7faa800b17d0 legacy=0x7faa801da890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:08.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:08.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 1 monclient: found mon.c 2024-03-29T10:53:08.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:08.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:08.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa800ad0b0 con 0x7faa801422b0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.351+0000 7faa767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 5 monclient: authenticate success, global_id 4466 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 10 monclient: _renew_subs 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa800ab490 con 0x7faa801422b0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 10 monclient: _renew_subs 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa800abe70 con 0x7faa801422b0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.352+0000 7faa87df4640 1 librados: init done 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa70004a90 con 0x7faa801422b0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7faa70004e60 con 0x7faa801422b0 2024-03-29T10:53:08.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa767fc640 10 monclient: dump: 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:08.337 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:08.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.353+0000 7faa777fe640 4 set_mon_vals no callback set 2024-03-29T10:53:08.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.355+0000 7faa87df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:08.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.355+0000 7faa87df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:08.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.355+0000 7faa87df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:08.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.355+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa68005080 con 0x7faa801422b0 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.355+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7faa70004fc0 con 0x7faa801422b0 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.356+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7faa70084c10 con 0x7faa801422b0 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.356+0000 7faa767fc640 10 client.4466.objecter ms_dispatch 0x7faa80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.356+0000 7faa767fc640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.356+0000 7faa767fc640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.356+0000 7faa767fc640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.356+0000 7faa85b90640 10 monclient: get_auth_request con 0x7faa6406dd00 auth_method 0 2024-03-29T10:53:08.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.357+0000 7faa767fc640 10 client.4466.objecter ms_handle_connect 0x7faa6406dd00 2024-03-29T10:53:08.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.363+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa700504f0 con 0x7faa801422b0 2024-03-29T10:53:08.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.363+0000 7faa767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:08.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.363+0000 7faa767fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:08.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.657+0000 7faa87df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:08.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.657+0000 7faa87df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:08.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.657+0000 7faa87df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:08.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.657+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa68005330 con 0x7faa801422b0 2024-03-29T10:53:08.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.657+0000 7faa767fc640 1 -- 172.21.15.79:0/307771462 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7faa70052420 con 0x7faa801422b0 2024-03-29T10:53:08.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.657+0000 7faa767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:08.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.658+0000 7faa767fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:08.639 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:08.640 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:08.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 10 librados: watch_flush enter 2024-03-29T10:53:08.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 10 librados: watch_flush exit 2024-03-29T10:53:08.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T10:53:08.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 10 client.4466.objecter successfully canceled tick 2024-03-29T10:53:08.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 >> v1:172.21.15.79:6816/4290694627 conn(0x7faa6406dd00 legacy=0x7faa640701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:08.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 10 monclient: shutdown 2024-03-29T10:53:08.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:08.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.661+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 >> v1:172.21.15.79:6790/0 conn(0x7faa801422b0 legacy=0x7faa8014d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:08.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.663+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 shutdown_connections 2024-03-29T10:53:08.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.663+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 >> 172.21.15.79:0/307771462 conn(0x7faa80066a00 msgr2=0x7faa8013c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:08.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.663+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 shutdown_connections 2024-03-29T10:53:08.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.663+0000 7faa87df4640 1 -- 172.21.15.79:0/307771462 wait complete. 2024-03-29T10:53:08.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:08.663+0000 7faa87df4640 1 librados: shutdown 2024-03-29T10:53:08.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:15.468 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.648+0000 7f397c7fb640 10 monclient: get_monmap_and_config 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.648+0000 7f397c7fb640 10 monclient: build_initial_monmap 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.648+0000 7f397c7fb640 10 monclient: monmap: 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:15.649793+0000 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:15.649793+0000 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:15.630 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:15.631 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:15.631 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:15.631 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:15.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.651+0000 7f397c7fb640 1 Processor -- start 2024-03-29T10:53:15.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.652+0000 7f397c7fb640 1 -- start start 2024-03-29T10:53:15.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.652+0000 7f397c7fb640 10 monclient: init 2024-03-29T10:53:15.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:15.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:15.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient(hunting): picked mon.noname-c con 0x7f397406c250 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:15.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient(hunting): picked mon.noname-b con 0x7f39740ae650 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:15.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient(hunting): picked mon.noname-a con 0x7f397413f8c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740adf50 con 0x7f397413f8c0 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740ae230 con 0x7f397406c250 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397413fd50 con 0x7f39740ae650 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.653+0000 7f397c7fb640 10 monclient(hunting): authenticate will time out at 1314.945471s 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.656+0000 7f3979d96640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f39740ae650 0x7f397413f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:53878/0 (socket says 172.21.15.79:53878) 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.656+0000 7f3979d96640 1 -- 172.21.15.79:0/2930873944 learned_addr learned my addr 172.21.15.79:0/2930873944 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.657+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148727279 0 0) 0x7f397413fd50 con 0x7f39740ae650 2024-03-29T10:53:15.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.657+0000 7f3979595640 10 monclient(hunting): _negotiate 2024-03-29T10:53:15.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.657+0000 7f3979595640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:15.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.657+0000 7f3979595640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:15.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.657+0000 7f3979595640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:53:15.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.657+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3960003520 con 0x7f39740ae650 2024-03-29T10:53:15.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.658+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480412006 0 0) 0x7f3960003520 con 0x7f39740ae650 2024-03-29T10:53:15.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.658+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 >> v1:172.21.15.82:6789/0 conn(0x7f397406c250 legacy=0x7f39741427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:15.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.660+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 >> v1:172.21.15.79:6789/0 conn(0x7f397413f8c0 legacy=0x7f397414db30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:15.642 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: _finish_hunting 0 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 1 monclient: found mon.noname-b 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397406db20 con 0x7f39740ae650 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: _finish_auth 0 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: _check_auth_tickets 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3970001670 con 0x7f39740ae650 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f3979595640 10 monclient: dump: 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:15.647 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f397c7fb640 5 monclient: authenticate success, global_id 4469 2024-03-29T10:53:15.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.661+0000 7f397c7fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.665+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3970001840 con 0x7f39740ae650 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.665+0000 7f3979595640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.665+0000 7f397c7fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.665+0000 7f397c7fb640 4 set_mon_vals no callback set 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.666+0000 7f397c7fb640 10 monclient: shutdown 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.666+0000 7f397c7fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.666+0000 7f397c7fb640 1 -- 172.21.15.79:0/2930873944 >> v1:172.21.15.79:6790/0 conn(0x7f39740ae650 legacy=0x7f397413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.666+0000 7f3979595640 1 -- 172.21.15.79:0/2930873944 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3970001c90 con 0x7f39740ae650 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.666+0000 7f3979595640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.667+0000 7f397c7fb640 1 -- 172.21.15.79:0/2930873944 shutdown_connections 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.667+0000 7f397c7fb640 1 -- 172.21.15.79:0/2930873944 >> 172.21.15.79:0/2930873944 conn(0x7f3974067610 msgr2=0x7f39740b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.667+0000 7f397c7fb640 1 -- 172.21.15.79:0/2930873944 shutdown_connections 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.667+0000 7f397c7fb640 1 -- 172.21.15.79:0/2930873944 wait complete. 2024-03-29T10:53:15.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.668+0000 7f397c7fb640 10 monclient: build_initial_monmap 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.668+0000 7f397c7fb640 10 monclient: monmap: 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 1 librados: starting msgr at 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 1 librados: starting objecter 2024-03-29T10:53:15.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 1 Processor -- start 2024-03-29T10:53:15.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 1 -- start start 2024-03-29T10:53:15.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 1 librados: setting wanted keys 2024-03-29T10:53:15.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 1 librados: calling monclient init 2024-03-29T10:53:15.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.669+0000 7f397c7fb640 10 monclient: init 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient(hunting): picked mon.noname-b con 0x7f397406c250 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient(hunting): picked mon.noname-c con 0x7f39740ae650 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient(hunting): picked mon.noname-a con 0x7f397413f8c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3974072790 con 0x7f397413f8c0 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3974071940 con 0x7f397406c250 2024-03-29T10:53:15.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39741ca6d0 con 0x7f39740ae650 2024-03-29T10:53:15.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:15.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.670+0000 7f397c7fb640 10 monclient(hunting): authenticate will time out at 1314.962234s 2024-03-29T10:53:15.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.674+0000 7f397ad98640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f397413f8c0 0x7f39741dae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:58915/0 (socket says 172.21.15.79:58915) 2024-03-29T10:53:15.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.674+0000 7f397ad98640 1 -- 172.21.15.79:0/3129934298 learned_addr learned my addr 172.21.15.79:0/3129934298 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:15.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.674+0000 7f396affd640 10 client.?.objecter ms_handle_connect 0x7f397413f8c0 2024-03-29T10:53:15.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.674+0000 7f396affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.675+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689623259 0 0) 0x7f3974072790 con 0x7f397413f8c0 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.675+0000 7f396affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.675+0000 7f396affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.675+0000 7f396affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.675+0000 7f396affd640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.675+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3958004eb0 con 0x7f397413f8c0 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 client.?.objecter ms_handle_connect 0x7f397406c250 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739099334 0 0) 0x7f3974071940 con 0x7f397406c250 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:15.660 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3974072790 con 0x7f397406c250 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918587864 0 0) 0x7f3974072790 con 0x7f397406c250 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3974071940 con 0x7f397406c250 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f396c001b10 con 0x7f397406c250 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): dump: 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:15.661 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913045162 0 0) 0x7f3958004eb0 con 0x7f397413f8c0 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3974072790 con 0x7f397413f8c0 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f39640018e0 con 0x7f397413f8c0 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.677+0000 7f396affd640 10 monclient(hunting): dump: 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:15.662 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3582048003 0 0) 0x7f3974071940 con 0x7f397406c250 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 >> v1:172.21.15.79:6790/0 conn(0x7f39740ae650 legacy=0x7f39741da730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 >> v1:172.21.15.79:6789/0 conn(0x7f397413f8c0 legacy=0x7f39741dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 monclient: found mon.b 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:15.663 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39741ca860 con 0x7f397406c250 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: _finish_auth 0 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f396c002020 con 0x7f397406c250 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f396c003b10 con 0x7f397406c250 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.678+0000 7f396affd640 10 monclient: dump: 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:15.664 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:15.665 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:15.665 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:15.666 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:15.666 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:15.666 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:15.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.679+0000 7f396bfff640 4 set_mon_vals no callback set 2024-03-29T10:53:15.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:53:15.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 10 monclient: _renew_subs 2024-03-29T10:53:15.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:15.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3974072130 con 0x7f397406c250 2024-03-29T10:53:15.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:15.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 10 monclient: _renew_subs 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39740724d0 con 0x7f397406c250 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.680+0000 7f397c7fb640 1 librados: init done 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.682+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f396c003c70 con 0x7f397406c250 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.683+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f396c083b30 con 0x7f397406c250 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.683+0000 7f396affd640 10 client.4399.objecter ms_dispatch 0x7f3974000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.683+0000 7f396affd640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.683+0000 7f396affd640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.683+0000 7f396affd640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.683+0000 7f3979d96640 10 monclient: get_auth_request con 0x7f395807ee90 auth_method 0 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.684+0000 7f397c7fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.684+0000 7f397c7fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.684+0000 7f397c7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.684+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3974141ea0 con 0x7f397406c250 2024-03-29T10:53:15.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.684+0000 7f396affd640 10 client.4399.objecter ms_handle_connect 0x7f395807ee90 2024-03-29T10:53:15.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.691+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f396c04f410 con 0x7f397406c250 2024-03-29T10:53:15.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.691+0000 7f396affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:15.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:15.691+0000 7f396affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:16.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.041+0000 7f397c7fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:16.022 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.041+0000 7f397c7fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:16.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.041+0000 7f397c7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:16.023 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.041+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39741ca9f0 con 0x7f397406c250 2024-03-29T10:53:16.024 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.043+0000 7f396affd640 1 -- 172.21.15.79:0/3129934298 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f396c051340 con 0x7f397406c250 2024-03-29T10:53:16.024 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.043+0000 7f396affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:16.024 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.043+0000 7f396affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:16.027 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:16.027 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:16.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 10 librados: watch_flush enter 2024-03-29T10:53:16.030 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 10 librados: watch_flush exit 2024-03-29T10:53:16.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T10:53:16.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 10 client.4399.objecter successfully canceled tick 2024-03-29T10:53:16.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 >> v1:172.21.15.79:6816/4290694627 conn(0x7f395807ee90 legacy=0x7f3958081350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:16.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 10 monclient: shutdown 2024-03-29T10:53:16.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:16.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.049+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 >> v1:172.21.15.82:6789/0 conn(0x7f397406c250 legacy=0x7f397413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:16.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.050+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 shutdown_connections 2024-03-29T10:53:16.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.050+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 >> 172.21.15.79:0/3129934298 conn(0x7f3974067610 msgr2=0x7f39741427b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:16.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.050+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 shutdown_connections 2024-03-29T10:53:16.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.050+0000 7f397c7fb640 1 -- 172.21.15.79:0/3129934298 wait complete. 2024-03-29T10:53:16.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:16.050+0000 7f397c7fb640 1 librados: shutdown 2024-03-29T10:53:16.048 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:22.855 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:23.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.024+0000 7f40be5a8640 10 monclient: get_monmap_and_config 2024-03-29T10:53:23.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.024+0000 7f40be5a8640 10 monclient: build_initial_monmap 2024-03-29T10:53:23.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.024+0000 7f40be5a8640 10 monclient: monmap: 2024-03-29T10:53:23.007 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:23.007 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:23.007 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:23.025504+0000 2024-03-29T10:53:23.008 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:23.025504+0000 2024-03-29T10:53:23.008 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:23.008 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.008 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:23.009 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:23.009 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:23.009 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.009 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.025+0000 7f40be5a8640 1 Processor -- start 2024-03-29T10:53:23.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.025+0000 7f40be5a8640 1 -- start start 2024-03-29T10:53:23.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.025+0000 7f40be5a8640 10 monclient: init 2024-03-29T10:53:23.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:23.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:23.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient(hunting): picked mon.noname-a con 0x7f40b81422b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:23.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient(hunting): picked mon.noname-b con 0x7f40b8142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:23.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient(hunting): picked mon.noname-c con 0x7f40b80b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:23.031 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b8142c80 con 0x7f40b81422b0 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b80aae40 con 0x7f40b80b17d0 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b813fbd0 con 0x7f40b8142690 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.026+0000 7f40be5a8640 10 monclient(hunting): authenticate will time out at 1322.319103s 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40bcb45640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f40b80b17d0 0x7f40b80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:44962/0 (socket says 172.21.15.79:44962) 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40bcb45640 1 -- 172.21.15.79:0/3627050987 learned_addr learned my addr 172.21.15.79:0/3627050987 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:23.032 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806097788 0 0) 0x7f40b80aae40 con 0x7f40b80b17d0 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40b6ffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.027+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a0003520 con 0x7f40b80b17d0 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147522731 0 0) 0x7f40b8142c80 con 0x7f40b81422b0 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:23.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b80aae40 con 0x7f40b81422b0 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007266954 0 0) 0x7f40b813fbd0 con 0x7f40b8142690 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:23.034 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b8142c80 con 0x7f40b8142690 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494200230 0 0) 0x7f40a0003520 con 0x7f40b80b17d0 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 >> v1:172.21.15.79:6790/0 conn(0x7f40b8142690 legacy=0x7f40b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 >> v1:172.21.15.79:6789/0 conn(0x7f40b81422b0 legacy=0x7f40b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 monclient: found mon.noname-c 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:53:23.035 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b8151940 con 0x7f40b80b17d0 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f40b0001990 con 0x7f40b80b17d0 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: dump: 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.036 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40be5a8640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40be5a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40b0001af0 con 0x7f40b80b17d0 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 1 -- 172.21.15.79:0/3627050987 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f40b0001f40 con 0x7f40b80b17d0 2024-03-29T10:53:23.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40b6ffd640 10 monclient: dump: 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:23.038 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40be5a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.028+0000 7f40be5a8640 4 set_mon_vals no callback set 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 10 monclient: shutdown 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 1 -- 172.21.15.79:0/3627050987 >> v1:172.21.15.82:6789/0 conn(0x7f40b80b17d0 legacy=0x7f40b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 1 -- 172.21.15.79:0/3627050987 shutdown_connections 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 1 -- 172.21.15.79:0/3627050987 >> 172.21.15.79:0/3627050987 conn(0x7f40b8066a00 msgr2=0x7f40b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 1 -- 172.21.15.79:0/3627050987 shutdown_connections 2024-03-29T10:53:23.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.029+0000 7f40be5a8640 1 -- 172.21.15.79:0/3627050987 wait complete. 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.030+0000 7f40be5a8640 10 monclient: build_initial_monmap 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.030+0000 7f40be5a8640 10 monclient: monmap: 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:23.040 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 1 librados: starting msgr at 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 1 librados: starting objecter 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 1 Processor -- start 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 1 -- start start 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 1 librados: setting wanted keys 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 1 librados: calling monclient init 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.031+0000 7f40be5a8640 10 monclient: init 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.032+0000 7f40be5a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.032+0000 7f40be5a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:23.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.032+0000 7f40be5a8640 10 monclient(hunting): picked mon.noname-c con 0x7f40b80b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.032+0000 7f40be5a8640 10 monclient(hunting): picked mon.noname-b con 0x7f40b81422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.032+0000 7f40be5a8640 10 monclient(hunting): picked mon.noname-a con 0x7f40b8142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40be5a8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b8131470 con 0x7f40b8142690 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40be5a8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b8131900 con 0x7f40b81422b0 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40be5a8640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b806ee80 con 0x7f40b80b17d0 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40be5a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40bcb45640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f40b8142690 0x7f40b81e6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:55124/0 (socket says 172.21.15.79:55124) 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40bcb45640 1 -- 172.21.15.79:0/774621022 learned_addr learned my addr 172.21.15.79:0/774621022 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:23.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40be5a8640 10 monclient(hunting): authenticate will time out at 1322.325459s 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 client.?.objecter ms_handle_connect 0x7f40b8142690 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996210544 0 0) 0x7f40b8131470 con 0x7f40b8142690 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a4004eb0 con 0x7f40b8142690 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 client.?.objecter ms_handle_connect 0x7f40b80b17d0 2024-03-29T10:53:23.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.033+0000 7f40b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:23.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859211526 0 0) 0x7f40b806ee80 con 0x7f40b80b17d0 2024-03-29T10:53:23.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:23.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:23.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:23.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:53:23.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b8131470 con 0x7f40b80b17d0 2024-03-29T10:53:23.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725039543 0 0) 0x7f40a4004eb0 con 0x7f40b8142690 2024-03-29T10:53:23.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40b806ee80 con 0x7f40b8142690 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3621293069 0 0) 0x7f40b8131470 con 0x7f40b80b17d0 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40a4004eb0 con 0x7f40b80b17d0 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f40b0001e00 con 0x7f40b8142690 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.034+0000 7f40b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:23.046 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 client.?.objecter ms_handle_connect 0x7f40b81422b0 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f40ac0018b0 con 0x7f40b80b17d0 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:23.047 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 445583285 0 0) 0x7f40b806ee80 con 0x7f40b8142690 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 >> v1:172.21.15.79:6790/0 conn(0x7f40b80b17d0 legacy=0x7f40b8131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.048 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 >> v1:172.21.15.82:6789/0 conn(0x7f40b81422b0 legacy=0x7f40b8070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 1 monclient: found mon.a 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b806f160 con 0x7f40b8142690 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.035+0000 7f40b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 5 monclient: authenticate success, global_id 4410 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 10 monclient: _renew_subs 2024-03-29T10:53:23.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b8131b50 con 0x7f40b8142690 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 10 monclient: _renew_subs 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b812fd80 con 0x7f40b8142690 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40be5a8640 1 librados: init done 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40b0001f60 con 0x7f40b8142690 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f40b0004fb0 con 0x7f40b8142690 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b67fc640 4 set_mon_vals no callback set 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:23.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.036+0000 7f40b4ff9640 10 monclient: dump: 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:23.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.038+0000 7f40be5a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.038+0000 7f40be5a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.038+0000 7f40be5a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.038+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4098005080 con 0x7f40b8142690 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.038+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f40b001af80 con 0x7f40b8142690 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b7fff640 10 monclient: get_auth_request con 0x7f40a406dee0 auth_method 0 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f40b0084e50 con 0x7f40b8142690 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b4ff9640 10 client.4410.objecter ms_dispatch 0x7f40b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b4ff9640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b4ff9640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b4ff9640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T10:53:23.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.039+0000 7f40b4ff9640 10 client.4410.objecter ms_handle_connect 0x7f40a406dee0 2024-03-29T10:53:23.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.051+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40b0050560 con 0x7f40b8142690 2024-03-29T10:53:23.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.051+0000 7f40b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:23.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.051+0000 7f40b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:23.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.459+0000 7f40be5a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:23.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.459+0000 7f40be5a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:23.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.459+0000 7f40be5a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:23.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.459+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4098005330 con 0x7f40b8142690 2024-03-29T10:53:23.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.461+0000 7f40b4ff9640 1 -- 172.21.15.79:0/774621022 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f40b0052660 con 0x7f40b8142690 2024-03-29T10:53:23.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.461+0000 7f40b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:23.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.461+0000 7f40b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:23.442 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:23.443 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:23.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.465+0000 7f40be5a8640 10 librados: watch_flush enter 2024-03-29T10:53:23.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.465+0000 7f40be5a8640 10 librados: watch_flush exit 2024-03-29T10:53:23.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.465+0000 7f40be5a8640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T10:53:23.447 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.465+0000 7f40be5a8640 10 client.4410.objecter successfully canceled tick 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.465+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 >> v1:172.21.15.79:6816/4290694627 conn(0x7f40a406dee0 legacy=0x7f40a40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 10 monclient: shutdown 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 >> v1:172.21.15.79:6789/0 conn(0x7f40b8142690 legacy=0x7f40b81e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 shutdown_connections 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 >> 172.21.15.79:0/774621022 conn(0x7f40b8066a00 msgr2=0x7f40b813dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 shutdown_connections 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 1 -- 172.21.15.79:0/774621022 wait complete. 2024-03-29T10:53:23.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:23.466+0000 7f40be5a8640 1 librados: shutdown 2024-03-29T10:53:23.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:30.271 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:30.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.442+0000 7fdbf399d640 10 monclient: get_monmap_and_config 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.442+0000 7fdbf399d640 10 monclient: build_initial_monmap 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.442+0000 7fdbf399d640 10 monclient: monmap: 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:30.443394+0000 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:30.443394+0000 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:30.424 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:30.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.443+0000 7fdbf399d640 1 Processor -- start 2024-03-29T10:53:30.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.444+0000 7fdbf399d640 1 -- start start 2024-03-29T10:53:30.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.444+0000 7fdbf399d640 10 monclient: init 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbec1422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbec142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbec0b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec142c80 con 0x7fdbec0b17d0 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0aae40 con 0x7fdbec1422b0 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec13fbd0 con 0x7fdbec142690 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:30.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.445+0000 7fdbf399d640 10 monclient(hunting): authenticate will time out at 1329.737622s 2024-03-29T10:53:30.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.447+0000 7fdbf0f38640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fdbec142690 0x7fdbec0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60468/0 (socket says 172.21.15.79:60468) 2024-03-29T10:53:30.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.447+0000 7fdbf1f3a640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fdbec0b17d0 0x7fdbec0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:36848/0 (socket says 172.21.15.79:36848) 2024-03-29T10:53:30.428 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.447+0000 7fdbf1f3a640 1 -- 172.21.15.79:0/2354678177 learned_addr learned my addr 172.21.15.79:0/2354678177 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:30.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027350320 0 0) 0x7fdbec142c80 con 0x7fdbec0b17d0 2024-03-29T10:53:30.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:30.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd8003520 con 0x7fdbec0b17d0 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139687874 0 0) 0x7fdbec13fbd0 con 0x7fdbec142690 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbec142c80 con 0x7fdbec142690 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1369650355 0 0) 0x7fdbd8003520 con 0x7fdbec0b17d0 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 >> v1:172.21.15.79:6790/0 conn(0x7fdbec142690 legacy=0x7fdbec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 >> v1:172.21.15.82:6789/0 conn(0x7fdbec1422b0 legacy=0x7fdbec13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:30.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 monclient: found mon.noname-a 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec151940 con 0x7fdbec0b17d0 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdbe4001a10 con 0x7fdbec0b17d0 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbe3fff640 10 monclient: dump: 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:30.436 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbf399d640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.448+0000 7fdbf399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbe4001bf0 con 0x7fdbec0b17d0 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbe3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbf399d640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbf399d640 4 set_mon_vals no callback set 2024-03-29T10:53:30.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbe3fff640 1 -- 172.21.15.79:0/2354678177 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdbe4001f80 con 0x7fdbec0b17d0 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbe3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.449+0000 7fdbe3fff640 10 monclient: dump: 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.450+0000 7fdbf399d640 10 monclient: shutdown 2024-03-29T10:53:30.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.450+0000 7fdbf399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.450+0000 7fdbf399d640 1 -- 172.21.15.79:0/2354678177 >> v1:172.21.15.79:6789/0 conn(0x7fdbec0b17d0 legacy=0x7fdbec0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.450+0000 7fdbf399d640 1 -- 172.21.15.79:0/2354678177 shutdown_connections 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.450+0000 7fdbf399d640 1 -- 172.21.15.79:0/2354678177 >> 172.21.15.79:0/2354678177 conn(0x7fdbec066a00 msgr2=0x7fdbec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.450+0000 7fdbf399d640 1 -- 172.21.15.79:0/2354678177 shutdown_connections 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.451+0000 7fdbf399d640 1 -- 172.21.15.79:0/2354678177 wait complete. 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.452+0000 7fdbf399d640 10 monclient: build_initial_monmap 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.452+0000 7fdbf399d640 10 monclient: monmap: 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:30.439 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.452+0000 7fdbf399d640 1 librados: starting msgr at 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.452+0000 7fdbf399d640 1 librados: starting objecter 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.453+0000 7fdbf399d640 1 Processor -- start 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.453+0000 7fdbf399d640 1 -- start start 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.453+0000 7fdbf399d640 1 librados: setting wanted keys 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.453+0000 7fdbf399d640 1 librados: calling monclient init 2024-03-29T10:53:30.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.453+0000 7fdbf399d640 10 monclient: init 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.454+0000 7fdbf399d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.454+0000 7fdbf399d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.454+0000 7fdbf399d640 10 monclient(hunting): picked mon.noname-b con 0x7fdbec0b17d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.454+0000 7fdbf399d640 10 monclient(hunting): picked mon.noname-a con 0x7fdbec070c60 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbf399d640 10 monclient(hunting): picked mon.noname-c con 0x7fdbec1391f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbf1f3a640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fdbec1391f0 0x7fdbec0b0a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43914/0 (socket says 172.21.15.79:43914) 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbf1f3a640 1 -- 172.21.15.79:0/2188766817 learned_addr learned my addr 172.21.15.79:0/2188766817 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec0abfd0 con 0x7fdbec070c60 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec1385c0 con 0x7fdbec0b17d0 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbe1ffb640 10 client.?.objecter ms_handle_connect 0x7fdbec1391f0 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:30.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbe1ffb640 10 client.?.objecter ms_handle_connect 0x7fdbec070c60 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22582474 0 0) 0x7fdbec0abfd0 con 0x7fdbec070c60 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.455+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbec06e2f0 con 0x7fdbec1391f0 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbf399d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbf399d640 10 monclient(hunting): authenticate will time out at 1329.748219s 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd0004eb0 con 0x7fdbec070c60 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218889506 0 0) 0x7fdbec06e2f0 con 0x7fdbec1391f0 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:53:30.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbec0abfd0 con 0x7fdbec1391f0 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992871987 0 0) 0x7fdbec0abfd0 con 0x7fdbec1391f0 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbec06e2f0 con 0x7fdbec1391f0 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdbe4001f00 con 0x7fdbec1391f0 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): dump: 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.450 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.456+0000 7fdbe1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1069677646 0 0) 0x7fdbd0004eb0 con 0x7fdbec070c60 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbec0abfd0 con 0x7fdbec070c60 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1601243650 0 0) 0x7fdbec06e2f0 con 0x7fdbec1391f0 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 >> v1:172.21.15.82:6789/0 conn(0x7fdbec0b17d0 legacy=0x7fdbec0717e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 >> v1:172.21.15.79:6789/0 conn(0x7fdbec070c60 legacy=0x7fdbec138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 monclient: found mon.c 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:30.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec071ef0 con 0x7fdbec1391f0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbe1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 10 monclient: _renew_subs 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbec0b1150 con 0x7fdbec1391f0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 10 monclient: _renew_subs 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbec1e8260 con 0x7fdbec1391f0 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.457+0000 7fdbf399d640 1 librados: init done 2024-03-29T10:53:30.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.458+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbe40015b0 con 0x7fdbec1391f0 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.458+0000 7fdbe1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.458+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdbe4004fb0 con 0x7fdbec1391f0 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.458+0000 7fdbe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.458+0000 7fdbe1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.458+0000 7fdbe1ffb640 10 monclient: dump: 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.453 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.459+0000 7fdbe2ffd640 4 set_mon_vals no callback set 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.460+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fdbe401af80 con 0x7fdbec1391f0 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.460+0000 7fdbf1739640 10 monclient: get_auth_request con 0x7fdbd006de70 auth_method 0 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.460+0000 7fdbf399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.461+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fdbe4084e60 con 0x7fdbec1391f0 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.461+0000 7fdbe1ffb640 10 client.4481.objecter ms_dispatch 0x7fdbec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.461+0000 7fdbe1ffb640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.461+0000 7fdbe1ffb640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:30.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.461+0000 7fdbe1ffb640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T10:53:30.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.462+0000 7fdbf399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:30.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.462+0000 7fdbf399d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:30.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.462+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc005080 con 0x7fdbec1391f0 2024-03-29T10:53:30.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.463+0000 7fdbe1ffb640 10 client.4481.objecter ms_handle_connect 0x7fdbd006de70 2024-03-29T10:53:30.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.470+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbe4050740 con 0x7fdbec1391f0 2024-03-29T10:53:30.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.470+0000 7fdbe1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:30.456 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.470+0000 7fdbe1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:30.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.795+0000 7fdbf399d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:30.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.795+0000 7fdbf399d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:30.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.795+0000 7fdbf399d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:30.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.795+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbcc005330 con 0x7fdbec1391f0 2024-03-29T10:53:30.778 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.796+0000 7fdbe1ffb640 1 -- 172.21.15.79:0/2188766817 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fdbe4052670 con 0x7fdbec1391f0 2024-03-29T10:53:30.778 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.796+0000 7fdbe1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:30.778 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.796+0000 7fdbe1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:30.781 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:30.781 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:30.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.802+0000 7fdbf399d640 10 librados: watch_flush enter 2024-03-29T10:53:30.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.802+0000 7fdbf399d640 10 librados: watch_flush exit 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.802+0000 7fdbf399d640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.802+0000 7fdbf399d640 10 client.4481.objecter successfully canceled tick 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 >> v1:172.21.15.79:6816/4290694627 conn(0x7fdbd006de70 legacy=0x7fdbd0070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 10 monclient: shutdown 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 >> v1:172.21.15.79:6790/0 conn(0x7fdbec1391f0 legacy=0x7fdbec0b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 shutdown_connections 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 >> 172.21.15.79:0/2188766817 conn(0x7fdbec066a00 msgr2=0x7fdbec13e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 shutdown_connections 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 -- 172.21.15.79:0/2188766817 wait complete. 2024-03-29T10:53:30.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:30.805+0000 7fdbf399d640 1 librados: shutdown 2024-03-29T10:53:30.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:37.612 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.776+0000 7fc7ab59e640 10 monclient: get_monmap_and_config 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.776+0000 7fc7ab59e640 10 monclient: build_initial_monmap 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.776+0000 7fc7ab59e640 10 monclient: monmap: 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:37.778325+0000 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:37.778325+0000 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:37.760 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.777+0000 7fc7ab59e640 1 Processor -- start 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.777+0000 7fc7ab59e640 1 -- start start 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.777+0000 7fc7ab59e640 10 monclient: init 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:37.761 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a4037230 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:37.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a40c4f20 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:37.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a4031e30 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:37.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40331f0 con 0x7fc7a4031e30 2024-03-29T10:53:37.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40310b0 con 0x7fc7a4037230 2024-03-29T10:53:37.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40c5320 con 0x7fc7a40c4f20 2024-03-29T10:53:37.762 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.778+0000 7fc7ab59e640 10 monclient(hunting): authenticate will time out at 1337.071565s 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a9d9b640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fc7a40c4f20 0x7fc7a4033610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60722/0 (socket says 172.21.15.79:60722) 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a9d9b640 1 -- 172.21.15.79:0/3919581213 learned_addr learned my addr 172.21.15.79:0/3919581213 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37298555 0 0) 0x7fc7a40c5320 con 0x7fc7a40c4f20 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:37.764 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:53:37.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc794003520 con 0x7fc7a40c4f20 2024-03-29T10:53:37.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236880001 0 0) 0x7fc794003520 con 0x7fc7a40c4f20 2024-03-29T10:53:37.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.779+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 >> v1:172.21.15.82:6789/0 conn(0x7fc7a4037230 legacy=0x7fc7a40c4810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:37.765 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.781+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 >> v1:172.21.15.79:6789/0 conn(0x7fc7a4031e30 legacy=0x7fc7a40d3100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: _finish_hunting 0 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 1 monclient: found mon.noname-b 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40073b0 con 0x7fc7a40c4f20 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: _finish_auth 0 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: _check_auth_tickets 2024-03-29T10:53:37.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7ab59e640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7ab59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc7a0001690 con 0x7fc7a40c4f20 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: dump: 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:37.767 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7ab59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 1 -- 172.21.15.79:0/3919581213 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7a0001920 con 0x7fc7a40c4f20 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7a959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7ab59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.784+0000 7fc7ab59e640 4 set_mon_vals no callback set 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.785+0000 7fc7ab59e640 10 monclient: shutdown 2024-03-29T10:53:37.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.785+0000 7fc7ab59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:37.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.785+0000 7fc7ab59e640 1 -- 172.21.15.79:0/3919581213 >> v1:172.21.15.79:6790/0 conn(0x7fc7a40c4f20 legacy=0x7fc7a4033610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:37.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.786+0000 7fc7ab59e640 1 -- 172.21.15.79:0/3919581213 shutdown_connections 2024-03-29T10:53:37.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.786+0000 7fc7ab59e640 1 -- 172.21.15.79:0/3919581213 >> 172.21.15.79:0/3919581213 conn(0x7fc7a403a400 msgr2=0x7fc7a4036850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:37.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.787+0000 7fc7ab59e640 1 -- 172.21.15.79:0/3919581213 shutdown_connections 2024-03-29T10:53:37.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.787+0000 7fc7ab59e640 1 -- 172.21.15.79:0/3919581213 wait complete. 2024-03-29T10:53:37.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.788+0000 7fc7ab59e640 10 monclient: build_initial_monmap 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.788+0000 7fc7ab59e640 10 monclient: monmap: 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.788+0000 7fc7ab59e640 1 librados: starting msgr at 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.788+0000 7fc7ab59e640 1 librados: starting objecter 2024-03-29T10:53:37.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.789+0000 7fc7ab59e640 1 Processor -- start 2024-03-29T10:53:37.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 1 -- start start 2024-03-29T10:53:37.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 1 librados: setting wanted keys 2024-03-29T10:53:37.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 1 librados: calling monclient init 2024-03-29T10:53:37.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient: init 2024-03-29T10:53:37.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:37.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a4031e30 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a4037230 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a40c4f20 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40c1570 con 0x7fc7a4037230 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40c7d00 con 0x7fc7a4031e30 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40c8b60 con 0x7fc7a40c4f20 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.791+0000 7fc7ab59e640 10 monclient(hunting): authenticate will time out at 1337.085039s 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc7aad9d640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fc7a40c4f20 0x7fc7a40c0e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:49814/0 (socket says 172.21.15.79:49814) 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc7aad9d640 1 -- 172.21.15.79:0/2258397310 learned_addr learned my addr 172.21.15.79:0/2258397310 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 10 client.?.objecter ms_handle_connect 0x7fc7a40c4f20 2024-03-29T10:53:37.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:37.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919811655 0 0) 0x7fc7a40c8b60 con 0x7fc7a40c4f20 2024-03-29T10:53:37.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:37.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:37.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:37.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:53:37.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.793+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c004eb0 con 0x7fc7a40c4f20 2024-03-29T10:53:37.778 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 10 client.?.objecter ms_handle_connect 0x7fc7a4037230 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597015208 0 0) 0x7fc78c004eb0 con 0x7fc7a40c4f20 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a40c8b60 con 0x7fc7a40c4f20 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc79c001950 con 0x7fc7a40c4f20 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.796+0000 7fc79affd640 10 monclient(hunting): dump: 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:37.779 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:37.780 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:37.780 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:37.780 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:37.780 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:37.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1669913767 0 0) 0x7fc7a40c8b60 con 0x7fc7a40c4f20 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 >> v1:172.21.15.82:6789/0 conn(0x7fc7a4031e30 legacy=0x7fc7a4164710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 >> v1:172.21.15.79:6789/0 conn(0x7fc7a4037230 legacy=0x7fc7a40c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 monclient: found mon.c 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40c8cf0 con 0x7fc7a40c4f20 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: _finish_auth 0 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc79c003610 con 0x7fc7a40c4f20 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:37.781 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc79c003a20 con 0x7fc7a40c4f20 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc79affd640 10 monclient: dump: 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:37.782 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:37.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc7a8d99640 4 set_mon_vals no callback set 2024-03-29T10:53:37.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc7ab59e640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc7ab59e640 10 monclient: _renew_subs 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc7ab59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.797+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a402f660 con 0x7fc7a40c4f20 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.798+0000 7fc7ab59e640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.798+0000 7fc7ab59e640 10 monclient: _renew_subs 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.798+0000 7fc7ab59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.798+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a402fa00 con 0x7fc7a40c4f20 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.798+0000 7fc7ab59e640 1 librados: init done 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.801+0000 7fc7ab59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.801+0000 7fc7ab59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.801+0000 7fc7ab59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.801+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a40c8310 con 0x7fc7a40c4f20 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.801+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fc79c003b80 con 0x7fc7a40c4f20 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.802+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fc79c084a70 con 0x7fc7a40c4f20 2024-03-29T10:53:37.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.802+0000 7fc79affd640 10 client.4487.objecter ms_dispatch 0x7fc7ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:37.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.802+0000 7fc79affd640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:37.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.802+0000 7fc79affd640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:37.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.802+0000 7fc79affd640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T10:53:37.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.803+0000 7fc7aa59c640 10 monclient: get_auth_request con 0x7fc78c06dd00 auth_method 0 2024-03-29T10:53:37.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.804+0000 7fc79affd640 10 client.4487.objecter ms_handle_connect 0x7fc78c06dd00 2024-03-29T10:53:37.792 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.810+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc79c0503b0 con 0x7fc7a40c4f20 2024-03-29T10:53:37.793 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.810+0000 7fc79affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:37.793 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:37.811+0000 7fc79affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:38.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.138+0000 7fc7ab59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:38.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.138+0000 7fc7ab59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:38.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.138+0000 7fc7ab59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:38.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.138+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7a40306c0 con 0x7fc7a40c4f20 2024-03-29T10:53:38.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.139+0000 7fc79affd640 1 -- 172.21.15.79:0/2258397310 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fc79c084120 con 0x7fc7a40c4f20 2024-03-29T10:53:38.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.139+0000 7fc79affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:38.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.139+0000 7fc79affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:38.122 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:38.122 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:38.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 10 librados: watch_flush enter 2024-03-29T10:53:38.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 10 librados: watch_flush exit 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 10 client.4487.objecter successfully canceled tick 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 >> v1:172.21.15.79:6816/4290694627 conn(0x7fc78c06dd00 legacy=0x7fc78c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 10 monclient: shutdown 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 >> v1:172.21.15.79:6790/0 conn(0x7fc7a40c4f20 legacy=0x7fc7a40c0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 shutdown_connections 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.142+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 >> 172.21.15.79:0/2258397310 conn(0x7fc7a403a400 msgr2=0x7fc7a40c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.143+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 shutdown_connections 2024-03-29T10:53:38.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.143+0000 7fc7ab59e640 1 -- 172.21.15.79:0/2258397310 wait complete. 2024-03-29T10:53:38.127 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:38.144+0000 7fc7ab59e640 1 librados: shutdown 2024-03-29T10:53:38.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:44.952 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:45.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.119+0000 7f7150829640 10 monclient: get_monmap_and_config 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.119+0000 7f7150829640 10 monclient: build_initial_monmap 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.119+0000 7f7150829640 10 monclient: monmap: 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:45.120331+0000 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:45.120331+0000 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:45.101 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.120+0000 7f7150829640 1 Processor -- start 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.120+0000 7f7150829640 1 -- start start 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.120+0000 7f7150829640 10 monclient: init 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient(hunting): picked mon.noname-a con 0x7f714c0b2800 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:45.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient(hunting): picked mon.noname-b con 0x7f714c0a9c00 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient(hunting): picked mon.noname-c con 0x7f714c0b38c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c141eb0 con 0x7f714c0b2800 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0b22d0 con 0x7f714c0b38c0 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c142d30 con 0x7f714c0a9c00 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.121+0000 7f7150829640 10 monclient(hunting): authenticate will time out at 1344.413519s 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.123+0000 7f7143fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f714c0a9c00 0x7f714c13bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43030/0 (socket says 172.21.15.79:43030) 2024-03-29T10:53:45.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.123+0000 7f7143fff640 1 -- 172.21.15.79:0/1385287918 learned_addr learned my addr 172.21.15.79:0/1385287918 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:45.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793348358 0 0) 0x7f714c141eb0 con 0x7f714c0b2800 2024-03-29T10:53:45.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): _negotiate 2024-03-29T10:53:45.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:45.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:45.106 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7138003520 con 0x7f714c0b2800 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176913647 0 0) 0x7f714c142d30 con 0x7f714c0a9c00 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): _negotiate 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:53:45.107 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c141eb0 con 0x7f714c0a9c00 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351412841 0 0) 0x7f7138003520 con 0x7f714c0b2800 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 >> v1:172.21.15.79:6790/0 conn(0x7f714c0a9c00 legacy=0x7f714c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 >> v1:172.21.15.82:6789/0 conn(0x7f714c0b38c0 legacy=0x7f714c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient: _finish_hunting 0 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 monclient: found mon.noname-a 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c0aaf90 con 0x7f714c0b2800 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient: _finish_auth 0 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient: _check_auth_tickets 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f7150829640 5 monclient: authenticate success, global_id 4425 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f7150829640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:45.108 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f713c001690 con 0x7f714c0b2800 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.125+0000 7f714a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f714a59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f714a59c640 10 monclient: dump: 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f7150829640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:45.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f714a59c640 1 -- 172.21.15.79:0/1385287918 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f713c0017f0 con 0x7f714c0b2800 2024-03-29T10:53:45.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f714a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:45.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f7150829640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:45.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.126+0000 7f7150829640 4 set_mon_vals no callback set 2024-03-29T10:53:45.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 10 monclient: shutdown 2024-03-29T10:53:45.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:45.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 1 -- 172.21.15.79:0/1385287918 >> v1:172.21.15.79:6789/0 conn(0x7f714c0b2800 legacy=0x7f714c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:45.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 1 -- 172.21.15.79:0/1385287918 shutdown_connections 2024-03-29T10:53:45.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 1 -- 172.21.15.79:0/1385287918 >> 172.21.15.79:0/1385287918 conn(0x7f714c066a00 msgr2=0x7f714c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:45.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 1 -- 172.21.15.79:0/1385287918 shutdown_connections 2024-03-29T10:53:45.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.127+0000 7f7150829640 1 -- 172.21.15.79:0/1385287918 wait complete. 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.128+0000 7f7150829640 10 monclient: build_initial_monmap 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.128+0000 7f7150829640 10 monclient: monmap: 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 1 librados: starting msgr at 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 1 librados: starting objecter 2024-03-29T10:53:45.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 1 Processor -- start 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 1 -- start start 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 1 librados: setting wanted keys 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 1 librados: calling monclient init 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.129+0000 7f7150829640 10 monclient: init 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient(hunting): picked mon.noname-b con 0x7f714c0a9c00 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient(hunting): picked mon.noname-c con 0x7f714c0b2800 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient(hunting): picked mon.noname-a con 0x7f714c0b38c0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0aca40 con 0x7f714c0b38c0 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c13c480 con 0x7f714c0a9c00 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c13c610 con 0x7f714c0b2800 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:45.113 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.130+0000 7f7150829640 10 monclient(hunting): authenticate will time out at 1344.422217s 2024-03-29T10:53:45.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.133+0000 7f714ad9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f714c0a9c00 0x7f714c1da250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:43399/0 (socket says 172.21.15.79:43399) 2024-03-29T10:53:45.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.133+0000 7f714ad9d640 1 -- 172.21.15.79:0/2795203861 learned_addr learned my addr 172.21.15.79:0/2795203861 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:45.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.133+0000 7f71437fe640 10 client.?.objecter ms_handle_connect 0x7f714c0a9c00 2024-03-29T10:53:45.118 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.133+0000 7f71437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 client.?.objecter ms_handle_connect 0x7f714c0b2800 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170243245 0 0) 0x7f714c13c480 con 0x7f714c0a9c00 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f712c004eb0 con 0x7f714c0a9c00 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4185085741 0 0) 0x7f712c004eb0 con 0x7f714c0a9c00 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f714c13c480 con 0x7f714c0a9c00 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f713c003c30 con 0x7f714c0a9c00 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:45.119 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): dump: 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.134+0000 7f71437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965579111 0 0) 0x7f714c13c610 con 0x7f714c0b2800 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:45.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f712c005680 con 0x7f714c0b2800 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1352191904 0 0) 0x7f714c13c480 con 0x7f714c0a9c00 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 >> v1:172.21.15.79:6790/0 conn(0x7f714c0b2800 legacy=0x7f714c1da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.135+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 >> v1:172.21.15.79:6789/0 conn(0x7f714c0b38c0 legacy=0x7f714c1e1590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 1 monclient: found mon.b 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c13e8f0 con 0x7f714c0a9c00 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f71437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 5 monclient: authenticate success, global_id 4414 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 10 monclient: _renew_subs 2024-03-29T10:53:45.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:45.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c13ebc0 con 0x7f714c0a9c00 2024-03-29T10:53:45.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:45.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 10 monclient: _renew_subs 2024-03-29T10:53:45.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:45.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c13f490 con 0x7f714c0a9c00 2024-03-29T10:53:45.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.136+0000 7f7150829640 1 librados: init done 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f713c0019e0 con 0x7f714c0a9c00 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f71437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f713c003c30 con 0x7f714c0a9c00 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f71437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f71437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f71437fe640 10 monclient: dump: 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:45.124 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.138+0000 7f7149d9b640 4 set_mon_vals no callback set 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.139+0000 7f7150829640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.139+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f713c003e50 con 0x7f714c0a9c00 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.139+0000 7f7150829640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.139+0000 7f7150829640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.139+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7130005080 con 0x7f714c0a9c00 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f713c084b40 con 0x7f714c0a9c00 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f71437fe640 10 client.4414.objecter ms_dispatch 0x7f714c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:45.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f71437fe640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:45.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f71437fe640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:45.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f71437fe640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T10:53:45.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f7143fff640 10 monclient: get_auth_request con 0x7f712c06e160 auth_method 0 2024-03-29T10:53:45.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.140+0000 7f71437fe640 10 client.4414.objecter ms_handle_connect 0x7f712c06e160 2024-03-29T10:53:45.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.147+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f713c050420 con 0x7f714c0a9c00 2024-03-29T10:53:45.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.147+0000 7f71437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:45.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.147+0000 7f71437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:45.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.476+0000 7f7150829640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:45.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.476+0000 7f7150829640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:45.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.476+0000 7f7150829640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:53:45.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.476+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7130005330 con 0x7f714c0a9c00 2024-03-29T10:53:45.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.477+0000 7f71437fe640 1 -- 172.21.15.79:0/2795203861 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f713c052350 con 0x7f714c0a9c00 2024-03-29T10:53:45.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.477+0000 7f71437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:45.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.477+0000 7f71437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:45.459 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:45.459 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:45.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.481+0000 7f7150829640 10 librados: watch_flush enter 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.481+0000 7f7150829640 10 librados: watch_flush exit 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.481+0000 7f7150829640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.481+0000 7f7150829640 10 client.4414.objecter successfully canceled tick 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.482+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 >> v1:172.21.15.79:6816/4290694627 conn(0x7f712c06e160 legacy=0x7f712c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.482+0000 7f7150829640 10 monclient: shutdown 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.482+0000 7f7150829640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:45.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.482+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 >> v1:172.21.15.82:6789/0 conn(0x7f714c0a9c00 legacy=0x7f714c1da250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:45.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.483+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 shutdown_connections 2024-03-29T10:53:45.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.483+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 >> 172.21.15.79:0/2795203861 conn(0x7f714c066a00 msgr2=0x7f714c0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:45.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.484+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 shutdown_connections 2024-03-29T10:53:45.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.485+0000 7f7150829640 1 -- 172.21.15.79:0/2795203861 wait complete. 2024-03-29T10:53:45.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:45.485+0000 7f7150829640 1 librados: shutdown 2024-03-29T10:53:45.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:52.293 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.458+0000 7fb661d04640 10 monclient: get_monmap_and_config 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.458+0000 7fb661d04640 10 monclient: build_initial_monmap 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.458+0000 7fb661d04640 10 monclient: monmap: 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:52.459784+0000 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:52.459784+0000 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:52.440 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.459+0000 7fb661d04640 1 Processor -- start 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.460+0000 7fb661d04640 1 -- start start 2024-03-29T10:53:52.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.460+0000 7fb661d04640 10 monclient: init 2024-03-29T10:53:52.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.461+0000 7fb661d04640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:52.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.461+0000 7fb661d04640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:52.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.461+0000 7fb661d04640 10 monclient(hunting): picked mon.noname-c con 0x7fb65c0b48a0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:52.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.461+0000 7fb661d04640 10 monclient(hunting): picked mon.noname-b con 0x7fb65c0b3f70 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:52.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.463+0000 7fb661d04640 10 monclient(hunting): picked mon.noname-a con 0x7fb65c13fb20 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:52.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb65b7fe640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fb65c0b48a0 0x7fb65c0b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:47895/0 (socket says 172.21.15.79:47895) 2024-03-29T10:53:52.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb65b7fe640 1 -- 172.21.15.79:0/863517643 learned_addr learned my addr 172.21.15.79:0/863517643 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:52.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c0b43b0 con 0x7fb65c13fb20 2024-03-29T10:53:52.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c0b0b90 con 0x7fb65c0b48a0 2024-03-29T10:53:52.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c0b19a0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb661d04640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.464+0000 7fb661d04640 10 monclient(hunting): authenticate will time out at 1351.756909s 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824880847 0 0) 0x7fb65c0b0b90 con 0x7fb65c0b48a0 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb644003520 con 0x7fb65c0b48a0 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106239536 0 0) 0x7fb65c0b43b0 con 0x7fb65c13fb20 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c0b0b90 con 0x7fb65c13fb20 2024-03-29T10:53:52.451 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244398899 0 0) 0x7fb65c0b19a0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c0b43b0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106422702 0 0) 0x7fb644003520 con 0x7fb65c0b48a0 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 >> v1:172.21.15.79:6790/0 conn(0x7fb65c0b3f70 legacy=0x7fb65c13f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 >> v1:172.21.15.79:6789/0 conn(0x7fb65c13fb20 legacy=0x7fb65c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c0af800 con 0x7fb65c0b48a0 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:52.452 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb64c001690 con 0x7fb65c0b48a0 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.465+0000 7fb65a7fc640 10 monclient: dump: 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:52.453 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.466+0000 7fb661d04640 5 monclient: authenticate success, global_id 4417 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.466+0000 7fb661d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb64c0017f0 con 0x7fb65c0b48a0 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb65a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb65a7fc640 1 -- 172.21.15.79:0/863517643 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb64c001c00 con 0x7fb65c0b48a0 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb65a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb65a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb65a7fc640 10 monclient: dump: 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.454 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb661d04640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.467+0000 7fb661d04640 4 set_mon_vals no callback set 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.468+0000 7fb661d04640 10 monclient: shutdown 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.468+0000 7fb661d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.468+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 >> v1:172.21.15.82:6789/0 conn(0x7fb65c0b48a0 legacy=0x7fb65c0b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.469+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 shutdown_connections 2024-03-29T10:53:52.455 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.469+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 >> 172.21.15.79:0/863517643 conn(0x7fb65c06c8a0 msgr2=0x7fb65c06cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:52.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.470+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 shutdown_connections 2024-03-29T10:53:52.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.470+0000 7fb661d04640 1 -- 172.21.15.79:0/863517643 wait complete. 2024-03-29T10:53:52.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.471+0000 7fb661d04640 10 monclient: build_initial_monmap 2024-03-29T10:53:52.457 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.471+0000 7fb661d04640 10 monclient: monmap: 2024-03-29T10:53:52.457 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:52.457 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 1 librados: starting msgr at 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 1 librados: starting objecter 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 1 Processor -- start 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 1 -- start start 2024-03-29T10:53:52.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 1 librados: setting wanted keys 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 1 librados: calling monclient init 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.472+0000 7fb661d04640 10 monclient: init 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.473+0000 7fb661d04640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.473+0000 7fb661d04640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.473+0000 7fb661d04640 10 monclient(hunting): picked mon.noname-c con 0x7fb65c0b3f70 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.473+0000 7fb65b7fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb65c0b3f70 0x7fb65c13a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:39862/0 (socket says 172.21.15.79:39862) 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.473+0000 7fb65b7fe640 1 -- 172.21.15.79:0/3844222107 learned_addr learned my addr 172.21.15.79:0/3844222107 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.474+0000 7fb643fff640 10 client.?.objecter ms_handle_connect 0x7fb65c0b3f70 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.474+0000 7fb643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.474+0000 7fb661d04640 10 monclient(hunting): picked mon.noname-b con 0x7fb65c0b48a0 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.475+0000 7fb661d04640 10 monclient(hunting): picked mon.noname-a con 0x7fb65c13fb20 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.475+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c1d75e0 con 0x7fb65c13fb20 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.475+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c1d7270 con 0x7fb65c0b48a0 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.475+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65c137030 con 0x7fb65c0b3f70 2024-03-29T10:53:52.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.475+0000 7fb661d04640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:52.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.475+0000 7fb661d04640 10 monclient(hunting): authenticate will time out at 1351.767835s 2024-03-29T10:53:52.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.479+0000 7fb643fff640 10 client.?.objecter ms_handle_connect 0x7fb65c0b48a0 2024-03-29T10:53:52.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.479+0000 7fb643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:52.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.479+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255135330 0 0) 0x7fb65c137030 con 0x7fb65c0b3f70 2024-03-29T10:53:52.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb648004eb0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801398971 0 0) 0x7fb65c1d7270 con 0x7fb65c0b48a0 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c137030 con 0x7fb65c0b48a0 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227832267 0 0) 0x7fb648004eb0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb65c1d7270 con 0x7fb65c0b3f70 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb64c001c00 con 0x7fb65c0b3f70 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:52.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): dump: 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10000681 0 0) 0x7fb65c137030 con 0x7fb65c0b48a0 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb648004eb0 con 0x7fb65c0b48a0 2024-03-29T10:53:52.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb650001730 con 0x7fb65c0b48a0 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 10 monclient(hunting): dump: 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2668706873 0 0) 0x7fb65c1d7270 con 0x7fb65c0b3f70 2024-03-29T10:53:52.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 >> v1:172.21.15.82:6789/0 conn(0x7fb65c0b48a0 legacy=0x7fb65c1d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:52.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.480+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 >> v1:172.21.15.79:6789/0 conn(0x7fb65c13fb20 legacy=0x7fb65c135e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:52.469 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 1 monclient: found mon.c 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c137cc0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.481+0000 7fb643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 10 monclient: _renew_subs 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65c137f70 con 0x7fb65c0b3f70 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:52.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 10 monclient: _renew_subs 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c136720 con 0x7fb65c0b3f70 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb661d04640 1 librados: init done 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb64c004a90 con 0x7fb65c0b3f70 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb643fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb64c004e60 con 0x7fb65c0b3f70 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb643fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb643fff640 10 monclient: dump: 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:52.471 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.485+0000 7fb659ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.486+0000 7fb661d04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.487+0000 7fb661d04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.487+0000 7fb661d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:52.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.487+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63c005080 con 0x7fb65c0b3f70 2024-03-29T10:53:52.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.491+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fb64c004fc0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.491+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fb64c084cc0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.491+0000 7fb643fff640 10 client.4499.objecter ms_dispatch 0x7fb65c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:52.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.491+0000 7fb643fff640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:52.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.491+0000 7fb643fff640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:52.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.491+0000 7fb643fff640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:53:52.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.494+0000 7fb65affd640 10 monclient: get_auth_request con 0x7fb64806ded0 auth_method 0 2024-03-29T10:53:52.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.494+0000 7fb643fff640 10 client.4499.objecter ms_handle_connect 0x7fb64806ded0 2024-03-29T10:53:52.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.500+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb64c0505a0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.500+0000 7fb643fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:52.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.500+0000 7fb643fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:52.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.825+0000 7fb661d04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:52.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.825+0000 7fb661d04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:52.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.825+0000 7fb661d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:53:52.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.825+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb63c005330 con 0x7fb65c0b3f70 2024-03-29T10:53:52.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.827+0000 7fb643fff640 1 -- 172.21.15.79:0/3844222107 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fb64c0524d0 con 0x7fb65c0b3f70 2024-03-29T10:53:52.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.827+0000 7fb643fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:52.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.827+0000 7fb643fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:52.808 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:53:52.808 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:52.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.830+0000 7fb661d04640 10 librados: watch_flush enter 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 10 librados: watch_flush exit 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 >> v1:172.21.15.79:6816/4290694627 conn(0x7fb64806ded0 legacy=0x7fb648070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 10 monclient: shutdown 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 >> v1:172.21.15.79:6790/0 conn(0x7fb65c0b3f70 legacy=0x7fb65c13a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 shutdown_connections 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 >> 172.21.15.79:0/3844222107 conn(0x7fb65c06c8a0 msgr2=0x7fb65c13f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:52.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.832+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 shutdown_connections 2024-03-29T10:53:52.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.833+0000 7fb661d04640 1 -- 172.21.15.79:0/3844222107 wait complete. 2024-03-29T10:53:52.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:52.833+0000 7fb661d04640 1 librados: shutdown 2024-03-29T10:53:52.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:59.639 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.809+0000 7f6938463640 10 monclient: get_monmap_and_config 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.810+0000 7f6938463640 10 monclient: build_initial_monmap 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.810+0000 7f6938463640 10 monclient: monmap: 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:53:59.810986+0000 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:53:59.810986+0000 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:59.792 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:59.793 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:59.793 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:53:59.793 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:53:59.793 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.811+0000 7f6938463640 1 Processor -- start 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.811+0000 7f6938463640 1 -- start start 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.811+0000 7f6938463640 10 monclient: init 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.812+0000 7f6938463640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.812+0000 7f6938463640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.812+0000 7f6938463640 10 monclient(hunting): picked mon.noname-b con 0x7f69300acf60 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:59.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 10 monclient(hunting): picked mon.noname-c con 0x7f693013ff40 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:59.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 10 monclient(hunting): picked mon.noname-a con 0x7f69301403f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:59.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300b4310 con 0x7f69301403f0 2024-03-29T10:53:59.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6930140800 con 0x7f693013ff40 2024-03-29T10:53:59.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300aec00 con 0x7f69300acf60 2024-03-29T10:53:59.795 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:59.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.813+0000 7f6938463640 10 monclient(hunting): authenticate will time out at 1359.106101s 2024-03-29T10:53:59.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.814+0000 7f69361ff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f69300acf60 0x7f6930139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:58996/0 (socket says 172.21.15.79:58996) 2024-03-29T10:53:59.796 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.814+0000 7f69361ff640 1 -- 172.21.15.79:0/1167232567 learned_addr learned my addr 172.21.15.79:0/1167232567 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071110675 0 0) 0x7f69300aec00 con 0x7f69300acf60 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c003520 con 0x7f69300acf60 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212891443 0 0) 0x7f691c003520 con 0x7f69300acf60 2024-03-29T10:53:59.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.815+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 >> v1:172.21.15.82:6789/0 conn(0x7f693013ff40 legacy=0x7f6930148840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:59.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.817+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 >> v1:172.21.15.79:6789/0 conn(0x7f69301403f0 legacy=0x7f693014bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:53:59.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:59.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 1 monclient: found mon.noname-b 2024-03-29T10:53:59.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300aeee0 con 0x7f69300acf60 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: _finish_auth 0 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f692c001690 con 0x7f69300acf60 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: dump: 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.800 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f6938463640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f6938463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f692c001860 con 0x7f69300acf60 2024-03-29T10:53:59.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 1 -- 172.21.15.79:0/1167232567 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f692c001cb0 con 0x7f69300acf60 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.818+0000 7f69351fd640 10 monclient: dump: 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:59.803 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.819+0000 7f6938463640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.819+0000 7f6938463640 4 set_mon_vals no callback set 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.820+0000 7f6938463640 10 monclient: shutdown 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.821+0000 7f6938463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.821+0000 7f6938463640 1 -- 172.21.15.79:0/1167232567 >> v1:172.21.15.79:6790/0 conn(0x7f69300acf60 legacy=0x7f6930139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.821+0000 7f6938463640 1 -- 172.21.15.79:0/1167232567 shutdown_connections 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.821+0000 7f6938463640 1 -- 172.21.15.79:0/1167232567 >> 172.21.15.79:0/1167232567 conn(0x7f69300b35e0 msgr2=0x7f69300b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.821+0000 7f6938463640 1 -- 172.21.15.79:0/1167232567 shutdown_connections 2024-03-29T10:53:59.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.821+0000 7f6938463640 1 -- 172.21.15.79:0/1167232567 wait complete. 2024-03-29T10:53:59.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.822+0000 7f6938463640 10 monclient: build_initial_monmap 2024-03-29T10:53:59.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.822+0000 7f6938463640 10 monclient: monmap: 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 1 librados: starting msgr at 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 1 librados: starting objecter 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 1 Processor -- start 2024-03-29T10:53:59.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 1 -- start start 2024-03-29T10:53:59.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 1 librados: setting wanted keys 2024-03-29T10:53:59.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 1 librados: calling monclient init 2024-03-29T10:53:59.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.823+0000 7f6938463640 10 monclient: init 2024-03-29T10:53:59.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.824+0000 7f6938463640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:59.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.824+0000 7f6938463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:59.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.824+0000 7f6938463640 10 monclient(hunting): picked mon.noname-b con 0x7f69300acf60 addr v1:172.21.15.82:6789/0 2024-03-29T10:53:59.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f69359fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f693013ff40 0x7f69301d5ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:58875/0 (socket says 172.21.15.79:58875) 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f69359fe640 1 -- 172.21.15.79:0/89553341 learned_addr learned my addr 172.21.15.79:0/89553341 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 10 monclient(hunting): picked mon.noname-a con 0x7f693013ff40 addr v1:172.21.15.79:6789/0 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 10 monclient(hunting): picked mon.noname-c con 0x7f69301403f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300aca00 con 0x7f693013ff40 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693013d2a0 con 0x7f69300acf60 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693013d430 con 0x7f69301403f0 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6938463640 10 monclient(hunting): authenticate will time out at 1359.119539s 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 client.?.objecter ms_handle_connect 0x7f693013ff40 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 client.?.objecter ms_handle_connect 0x7f69301403f0 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 client.?.objecter ms_handle_connect 0x7f69300acf60 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:59.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049428017 0 0) 0x7f69300aca00 con 0x7f693013ff40 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6914004eb0 con 0x7f693013ff40 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624119421 0 0) 0x7f693013d430 con 0x7f69301403f0 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.827+0000 7f6926ffd640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:53:59.813 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69300aca00 con 0x7f69301403f0 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613182471 0 0) 0x7f6914004eb0 con 0x7f693013ff40 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f693013d430 con 0x7f693013ff40 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6920001870 con 0x7f693013ff40 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:59.814 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124838411 0 0) 0x7f693013d2a0 con 0x7f69300acf60 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6914004eb0 con 0x7f69300acf60 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1125729187 0 0) 0x7f693013d430 con 0x7f693013ff40 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 >> v1:172.21.15.79:6790/0 conn(0x7f69301403f0 legacy=0x7f69301dd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 >> v1:172.21.15.82:6789/0 conn(0x7f69300acf60 legacy=0x7f69300ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:59.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 monclient: found mon.a 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300abc80 con 0x7f693013ff40 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 10 monclient: _renew_subs 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69301d6700 con 0x7f693013ff40 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 10 monclient: _renew_subs 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:59.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f693013a700 con 0x7f693013ff40 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6938463640 1 librados: init done 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69200019d0 con 0x7f693013ff40 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6920004aa0 con 0x7f693013ff40 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.828+0000 7f6926ffd640 10 monclient: dump: 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:53:59.817 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:53:59.818 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:53:59.818 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:53:59.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.829+0000 7f6927fff640 4 set_mon_vals no callback set 2024-03-29T10:53:59.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.830+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f6920004c00 con 0x7f693013ff40 2024-03-29T10:53:59.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.831+0000 7f6938463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:59.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.831+0000 7f6938463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:59.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.831+0000 7f6938463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:53:59.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.831+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6918005080 con 0x7f693013ff40 2024-03-29T10:53:59.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f69361ff640 10 monclient: get_auth_request con 0x7f6914076470 auth_method 0 2024-03-29T10:53:59.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f6926ffd640 10 client.4431.objecter ms_handle_connect 0x7f6914076470 2024-03-29T10:53:59.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f6920084a30 con 0x7f693013ff40 2024-03-29T10:53:59.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f6926ffd640 10 client.4431.objecter ms_dispatch 0x7f6930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:59.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f6926ffd640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:59.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f6926ffd640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:59.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.832+0000 7f6926ffd640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T10:53:59.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.840+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6920050310 con 0x7f693013ff40 2024-03-29T10:53:59.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.840+0000 7f6926ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:59.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:53:59.840+0000 7f6926ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:00.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.168+0000 7f6938463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:00.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.168+0000 7f6938463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:00.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.168+0000 7f6938463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:00.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.168+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6918005330 con 0x7f693013ff40 2024-03-29T10:54:00.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.169+0000 7f6926ffd640 1 -- 172.21.15.79:0/89553341 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f6920052240 con 0x7f693013ff40 2024-03-29T10:54:00.154 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:00.155 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:00.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.169+0000 7f6926ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:00.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.169+0000 7f6926ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:00.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 10 librados: watch_flush enter 2024-03-29T10:54:00.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 10 librados: watch_flush exit 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 10 client.4431.objecter successfully canceled tick 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 >> v1:172.21.15.79:6816/4290694627 conn(0x7f6914076470 legacy=0x7f6914078930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 10 monclient: shutdown 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 >> v1:172.21.15.79:6789/0 conn(0x7f693013ff40 legacy=0x7f69301d5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 shutdown_connections 2024-03-29T10:54:00.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.173+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 >> 172.21.15.79:0/89553341 conn(0x7f69300b35e0 msgr2=0x7f69300b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:00.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.174+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 shutdown_connections 2024-03-29T10:54:00.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.175+0000 7f6938463640 1 -- 172.21.15.79:0/89553341 wait complete. 2024-03-29T10:54:00.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:00.175+0000 7f6938463640 1 librados: shutdown 2024-03-29T10:54:00.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:06.983 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.153+0000 7fec4610e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.153+0000 7fec4610e640 10 monclient: build_initial_monmap 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.153+0000 7fec4610e640 10 monclient: monmap: 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:07.154649+0000 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:07.154649+0000 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.135 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:07.136 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:07.136 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:07.136 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.154+0000 7fec4610e640 1 Processor -- start 2024-03-29T10:54:07.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.154+0000 7fec4610e640 1 -- start start 2024-03-29T10:54:07.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.154+0000 7fec4610e640 10 monclient: init 2024-03-29T10:54:07.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.155+0000 7fec4610e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:07.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.155+0000 7fec4610e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:07.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 10 monclient(hunting): picked mon.noname-c con 0x7fec400b0770 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:07.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 10 monclient(hunting): picked mon.noname-a con 0x7fec4014a700 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:07.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 10 monclient(hunting): picked mon.noname-b con 0x7fec400ac5a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:07.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400aba70 con 0x7fec4014a700 2024-03-29T10:54:07.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400aecb0 con 0x7fec400b0770 2024-03-29T10:54:07.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400af150 con 0x7fec400ac5a0 2024-03-29T10:54:07.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:07.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec4610e640 10 monclient(hunting): authenticate will time out at 1366.448409s 2024-03-29T10:54:07.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec3effd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fec4014a700 0x7fec400abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:41538/0 (socket says 172.21.15.79:41538) 2024-03-29T10:54:07.138 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.156+0000 7fec3effd640 1 -- 172.21.15.79:0/583319522 learned_addr learned my addr 172.21.15.79:0/583319522 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:07.140 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656508417 0 0) 0x7fec400aecb0 con 0x7fec400b0770 2024-03-29T10:54:07.140 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec28003520 con 0x7fec400b0770 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593816861 0 0) 0x7fec400aba70 con 0x7fec4014a700 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec400aecb0 con 0x7fec4014a700 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716510280 0 0) 0x7fec400af150 con 0x7fec400ac5a0 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:07.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec400aba70 con 0x7fec400ac5a0 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 69523218 0 0) 0x7fec28003520 con 0x7fec400b0770 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 >> v1:172.21.15.79:6790/0 conn(0x7fec400ac5a0 legacy=0x7fec40151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 >> v1:172.21.15.79:6789/0 conn(0x7fec4014a700 legacy=0x7fec400abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400ac9d0 con 0x7fec400b0770 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:07.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fec300016b0 con 0x7fec400b0770 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.157+0000 7fec3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 10 monclient: dump: 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec4610e640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec4610e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec30001810 con 0x7fec400b0770 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 1 -- 172.21.15.79:0/583319522 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fec30001be0 con 0x7fec400b0770 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec4610e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec4610e640 4 set_mon_vals no callback set 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.158+0000 7fec3e7fc640 10 monclient: dump: 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:07.144 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 10 monclient: shutdown 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 1 -- 172.21.15.79:0/583319522 >> v1:172.21.15.82:6789/0 conn(0x7fec400b0770 legacy=0x7fec40149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 1 -- 172.21.15.79:0/583319522 shutdown_connections 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 1 -- 172.21.15.79:0/583319522 >> 172.21.15.79:0/583319522 conn(0x7fec4006b660 msgr2=0x7fec4006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 1 -- 172.21.15.79:0/583319522 shutdown_connections 2024-03-29T10:54:07.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.159+0000 7fec4610e640 1 -- 172.21.15.79:0/583319522 wait complete. 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.160+0000 7fec4610e640 10 monclient: build_initial_monmap 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.160+0000 7fec4610e640 10 monclient: monmap: 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:07.149 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.160+0000 7fec4610e640 1 librados: starting msgr at 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.160+0000 7fec4610e640 1 librados: starting objecter 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.161+0000 7fec4610e640 1 Processor -- start 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.161+0000 7fec4610e640 1 -- start start 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.161+0000 7fec4610e640 1 librados: setting wanted keys 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.161+0000 7fec4610e640 1 librados: calling monclient init 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.161+0000 7fec4610e640 10 monclient: init 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient(hunting): picked mon.noname-c con 0x7fec400ac5a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient(hunting): picked mon.noname-b con 0x7fec400b0770 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient(hunting): picked mon.noname-a con 0x7fec4014a700 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec40126120 con 0x7fec4014a700 2024-03-29T10:54:07.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec40125d60 con 0x7fec400b0770 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec40125240 con 0x7fec400ac5a0 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec4610e640 10 monclient(hunting): authenticate will time out at 1366.454579s 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec3effd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fec400b0770 0x7fec401265b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:52746/0 (socket says 172.21.15.79:52746) 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec3f7fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fec400ac5a0 0x7fec40144ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:40646/0 (socket says 172.21.15.79:40646) 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec3effd640 1 -- 172.21.15.79:0/2355031098 learned_addr learned my addr 172.21.15.79:0/2355031098 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec27fff640 10 client.?.objecter ms_handle_connect 0x7fec400b0770 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec27fff640 10 client.?.objecter ms_handle_connect 0x7fec400ac5a0 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec27fff640 10 client.?.objecter ms_handle_connect 0x7fec4014a700 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.162+0000 7fec27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4770261 0 0) 0x7fec40125240 con 0x7fec400ac5a0 2024-03-29T10:54:07.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c004eb0 con 0x7fec400ac5a0 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055253761 0 0) 0x7fec40125d60 con 0x7fec400b0770 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec40125240 con 0x7fec400b0770 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131384633 0 0) 0x7fec40126120 con 0x7fec4014a700 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:07.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec40125d60 con 0x7fec4014a700 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378546661 0 0) 0x7fec2c004eb0 con 0x7fec400ac5a0 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec40126120 con 0x7fec400ac5a0 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fec30001b20 con 0x7fec400ac5a0 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): dump: 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:07.153 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 536742697 0 0) 0x7fec40125240 con 0x7fec400b0770 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec2c004eb0 con 0x7fec400b0770 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fec340018d0 con 0x7fec400b0770 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.163+0000 7fec27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient(hunting): dump: 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:07.154 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1430778623 0 0) 0x7fec40125d60 con 0x7fec4014a700 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec40125240 con 0x7fec4014a700 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fec38001d90 con 0x7fec4014a700 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:07.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient(hunting): dump: 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3606461071 0 0) 0x7fec40125240 con 0x7fec4014a700 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 >> v1:172.21.15.79:6790/0 conn(0x7fec400ac5a0 legacy=0x7fec40144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.156 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 >> v1:172.21.15.82:6789/0 conn(0x7fec400b0770 legacy=0x7fec401265b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 monclient: found mon.a 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec401253d0 con 0x7fec4014a700 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 5 monclient: authenticate success, global_id 4437 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 10 monclient: _renew_subs 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec401dc140 con 0x7fec4014a700 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec380018f0 con 0x7fec4014a700 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:07.157 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fec38003b10 con 0x7fec4014a700 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec27fff640 10 monclient: dump: 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:07.158 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 10 monclient: _renew_subs 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec401457e0 con 0x7fec4014a700 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec4610e640 1 librados: init done 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.164+0000 7fec3dffb640 4 set_mon_vals no callback set 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.166+0000 7fec4610e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.166+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fec38019ae0 con 0x7fec4014a700 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.166+0000 7fec4610e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec4610e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400ab560 con 0x7fec4014a700 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fec38083a50 con 0x7fec4014a700 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec27fff640 10 client.4437.objecter ms_dispatch 0x7fec40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec27fff640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:07.159 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec27fff640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:07.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec27fff640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T10:54:07.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec3f7fe640 10 monclient: get_auth_request con 0x7fec2c06e150 auth_method 0 2024-03-29T10:54:07.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.167+0000 7fec27fff640 10 client.4437.objecter ms_handle_connect 0x7fec2c06e150 2024-03-29T10:54:07.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.175+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec3804f330 con 0x7fec4014a700 2024-03-29T10:54:07.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.175+0000 7fec27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:07.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.175+0000 7fec27fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:07.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.566+0000 7fec4610e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:07.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.566+0000 7fec4610e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:07.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.566+0000 7fec4610e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:07.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.566+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec40125ef0 con 0x7fec4014a700 2024-03-29T10:54:07.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.568+0000 7fec27fff640 1 -- 172.21.15.79:0/2355031098 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fec40125ef0 con 0x7fec4014a700 2024-03-29T10:54:07.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.568+0000 7fec27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:07.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.568+0000 7fec27fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:07.549 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:07.550 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 10 librados: watch_flush enter 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 10 librados: watch_flush exit 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 10 client.4437.objecter successfully canceled tick 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 >> v1:172.21.15.79:6816/4290694627 conn(0x7fec2c06e150 legacy=0x7fec2c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 10 monclient: shutdown 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 >> v1:172.21.15.79:6789/0 conn(0x7fec4014a700 legacy=0x7fec401e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:07.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 shutdown_connections 2024-03-29T10:54:07.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 >> 172.21.15.79:0/2355031098 conn(0x7fec4006b660 msgr2=0x7fec400b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:07.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 shutdown_connections 2024-03-29T10:54:07.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 -- 172.21.15.79:0/2355031098 wait complete. 2024-03-29T10:54:07.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:07.572+0000 7fec4610e640 1 librados: shutdown 2024-03-29T10:54:07.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:14.376 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:14.525 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.544+0000 7f8e27164640 10 monclient: get_monmap_and_config 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.544+0000 7f8e27164640 10 monclient: build_initial_monmap 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.544+0000 7f8e27164640 10 monclient: monmap: 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:14.545707+0000 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:14.545707+0000 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:14.526 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.546+0000 7f8e27164640 1 Processor -- start 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.546+0000 7f8e27164640 1 -- start start 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.546+0000 7f8e27164640 10 monclient: init 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.546+0000 7f8e27164640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.546+0000 7f8e27164640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.546+0000 7f8e27164640 10 monclient(hunting): picked mon.noname-b con 0x7f8e200b1f60 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:14.529 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.548+0000 7f8e24f00640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f8e200b1f60 0x7f8e20143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:37706/0 (socket says 172.21.15.79:37706) 2024-03-29T10:54:14.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e1ffff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f8e20071eb0 0x7f8e20146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:57819/0 (socket says 172.21.15.79:57819) 2024-03-29T10:54:14.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 10 monclient(hunting): picked mon.noname-a con 0x7f8e20071eb0 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:14.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 10 monclient(hunting): picked mon.noname-c con 0x7f8e20143d10 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:14.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2006ef80 con 0x7f8e20071eb0 2024-03-29T10:54:14.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2006f260 con 0x7f8e20143d10 2024-03-29T10:54:14.530 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200b40c0 con 0x7f8e200b1f60 2024-03-29T10:54:14.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:14.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.549+0000 7f8e27164640 10 monclient(hunting): authenticate will time out at 1373.841649s 2024-03-29T10:54:14.531 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e24f00640 1 -- 172.21.15.79:0/935467469 learned_addr learned my addr 172.21.15.79:0/935467469 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530016425 0 0) 0x7f8e200b40c0 con 0x7f8e200b1f60 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e1f7fe640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.550+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e10003520 con 0x7f8e200b1f60 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.551+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046231942 0 0) 0x7f8e10003520 con 0x7f8e200b1f60 2024-03-29T10:54:14.532 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.551+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 >> v1:172.21.15.82:6789/0 conn(0x7f8e20143d10 legacy=0x7f8e20146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 >> v1:172.21.15.79:6789/0 conn(0x7f8e20071eb0 legacy=0x7f8e20146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e20155d70 con 0x7f8e200b1f60 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8e0c001690 con 0x7f8e200b1f60 2024-03-29T10:54:14.536 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.554+0000 7f8e1f7fe640 10 monclient: dump: 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e0c001920 con 0x7f8e200b1f60 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:14.537 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e1f7fe640 1 -- 172.21.15.79:0/935467469 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8e0c001d30 con 0x7f8e200b1f60 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e1f7fe640 10 monclient: dump: 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e27164640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:54:14.538 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e27164640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:14.539 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.555+0000 7f8e27164640 4 set_mon_vals no callback set 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 10 monclient: shutdown 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 1 -- 172.21.15.79:0/935467469 >> v1:172.21.15.79:6790/0 conn(0x7f8e200b1f60 legacy=0x7f8e20143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 1 -- 172.21.15.79:0/935467469 shutdown_connections 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 1 -- 172.21.15.79:0/935467469 >> 172.21.15.79:0/935467469 conn(0x7f8e20066a00 msgr2=0x7f8e20066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 1 -- 172.21.15.79:0/935467469 shutdown_connections 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.556+0000 7f8e27164640 1 -- 172.21.15.79:0/935467469 wait complete. 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.557+0000 7f8e27164640 10 monclient: build_initial_monmap 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.557+0000 7f8e27164640 10 monclient: monmap: 2024-03-29T10:54:14.540 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 1 librados: starting msgr at 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 1 librados: starting objecter 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 1 Processor -- start 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 1 -- start start 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 1 librados: setting wanted keys 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 1 librados: calling monclient init 2024-03-29T10:54:14.541 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.558+0000 7f8e27164640 10 monclient: init 2024-03-29T10:54:14.542 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:14.542 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:14.542 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 10 monclient(hunting): picked mon.noname-b con 0x7f8e200b1f60 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:14.542 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 10 monclient(hunting): picked mon.noname-a con 0x7f8e20143d10 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:14.542 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 10 monclient(hunting): picked mon.noname-c con 0x7f8e20154030 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:14.542 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200aade0 con 0x7f8e20143d10 2024-03-29T10:54:14.543 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200ac1d0 con 0x7f8e200b1f60 2024-03-29T10:54:14.543 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e20140720 con 0x7f8e20154030 2024-03-29T10:54:14.543 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.559+0000 7f8e27164640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:14.543 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.560+0000 7f8e27164640 10 monclient(hunting): authenticate will time out at 1373.852109s 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e25701640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f8e20154030 0x7f8e20140010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:41106/0 (socket says 172.21.15.79:41106) 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1ffff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f8e20143d10 0x7f8e2013f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:59709/0 (socket says 172.21.15.79:59709) 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e25701640 1 -- 172.21.15.79:0/1585364237 learned_addr learned my addr 172.21.15.79:0/1585364237 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 client.?.objecter ms_handle_connect 0x7f8e20154030 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 client.?.objecter ms_handle_connect 0x7f8e20143d10 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209940541 0 0) 0x7f8e200aade0 con 0x7f8e20143d10 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:14.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04004eb0 con 0x7f8e20143d10 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497071168 0 0) 0x7f8e20140720 con 0x7f8e20154030 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.563+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e200aade0 con 0x7f8e20154030 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 client.?.objecter ms_handle_connect 0x7f8e200b1f60 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714865606 0 0) 0x7f8e04004eb0 con 0x7f8e20143d10 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e20140720 con 0x7f8e20143d10 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8e14001830 con 0x7f8e20143d10 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:14.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): dump: 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283738876 0 0) 0x7f8e200aade0 con 0x7f8e20154030 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e04004eb0 con 0x7f8e20154030 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8e18001ad0 con 0x7f8e20154030 2024-03-29T10:54:14.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.564+0000 7f8e1d7fa640 10 monclient(hunting): dump: 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:14.550 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506988375 0 0) 0x7f8e200ac1d0 con 0x7f8e200b1f60 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e200aade0 con 0x7f8e200b1f60 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3372658764 0 0) 0x7f8e20140720 con 0x7f8e20143d10 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 >> v1:172.21.15.79:6790/0 conn(0x7f8e20154030 legacy=0x7f8e20140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 >> v1:172.21.15.82:6789/0 conn(0x7f8e200b1f60 legacy=0x7f8e20153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 monclient: found mon.a 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200ac4b0 con 0x7f8e20143d10 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:14.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e14001d40 con 0x7f8e20143d10 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.567+0000 7f8e1effd640 4 set_mon_vals no callback set 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8e14004980 con 0x7f8e20143d10 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e1d7fa640 10 monclient: dump: 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:14.554 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e27164640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e27164640 10 monclient: _renew_subs 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e27164640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.568+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e200ab270 con 0x7f8e20143d10 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.569+0000 7f8e27164640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.569+0000 7f8e27164640 10 monclient: _renew_subs 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.569+0000 7f8e27164640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.569+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e200aa230 con 0x7f8e20143d10 2024-03-29T10:54:14.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.569+0000 7f8e27164640 1 librados: init done 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.571+0000 7f8e27164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.571+0000 7f8e27164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f8e1401a950 con 0x7f8e20143d10 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.571+0000 7f8e27164640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e20075f10 con 0x7f8e20143d10 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f8e140848f0 con 0x7f8e20143d10 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e1d7fa640 10 client.4443.objecter ms_dispatch 0x7f8e20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e1d7fa640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e1d7fa640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:14.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e1d7fa640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T10:54:14.558 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.572+0000 7f8e24f00640 10 monclient: get_auth_request con 0x7f8e0406dfa0 auth_method 0 2024-03-29T10:54:14.558 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.573+0000 7f8e1d7fa640 10 client.4443.objecter ms_handle_connect 0x7f8e0406dfa0 2024-03-29T10:54:14.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.581+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e14050000 con 0x7f8e20143d10 2024-03-29T10:54:14.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.581+0000 7f8e1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:14.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.581+0000 7f8e1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:14.888 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.907+0000 7f8e27164640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:14.888 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.907+0000 7f8e27164640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:14.888 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.907+0000 7f8e27164640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:14.889 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.907+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e200aaf90 con 0x7f8e20143d10 2024-03-29T10:54:14.893 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.911+0000 7f8e1d7fa640 1 -- 172.21.15.79:0/1585364237 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f8e200aaf90 con 0x7f8e20143d10 2024-03-29T10:54:14.893 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.911+0000 7f8e1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:14.893 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.911+0000 7f8e1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:14.894 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:14.894 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:14.895 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 10 librados: watch_flush enter 2024-03-29T10:54:14.895 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 10 librados: watch_flush exit 2024-03-29T10:54:14.895 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 10 client.4443.objecter successfully canceled tick 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 >> v1:172.21.15.79:6816/4290694627 conn(0x7f8e0406dfa0 legacy=0x7f8e04070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 10 monclient: shutdown 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 >> v1:172.21.15.79:6789/0 conn(0x7f8e20143d10 legacy=0x7f8e2013f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 shutdown_connections 2024-03-29T10:54:14.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.914+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 >> 172.21.15.79:0/1585364237 conn(0x7f8e20066a00 msgr2=0x7f8e200b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:14.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.915+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 shutdown_connections 2024-03-29T10:54:14.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.915+0000 7f8e27164640 1 -- 172.21.15.79:0/1585364237 wait complete. 2024-03-29T10:54:14.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:14.915+0000 7f8e27164640 1 librados: shutdown 2024-03-29T10:54:14.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:21.719 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:21.888 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.906+0000 7f4bf73c3640 10 monclient: get_monmap_and_config 2024-03-29T10:54:21.888 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.906+0000 7f4bf73c3640 10 monclient: build_initial_monmap 2024-03-29T10:54:21.888 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.906+0000 7f4bf73c3640 10 monclient: monmap: 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:21.908433+0000 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:21.908433+0000 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:21.889 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:21.892 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.910+0000 7f4bf73c3640 1 Processor -- start 2024-03-29T10:54:21.892 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.910+0000 7f4bf73c3640 1 -- start start 2024-03-29T10:54:21.892 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.910+0000 7f4bf73c3640 10 monclient: init 2024-03-29T10:54:21.896 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient(hunting): picked mon.noname-a con 0x7f4be8006e30 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient(hunting): picked mon.noname-c con 0x7f4be8035e00 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient(hunting): picked mon.noname-b con 0x7f4be80c9670 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be8035970 con 0x7f4be8006e30 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be8036280 con 0x7f4be8035e00 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be8035640 con 0x7f4be80c9670 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:21.897 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.911+0000 7f4bf73c3640 10 monclient(hunting): authenticate will time out at 1381.204911s 2024-03-29T10:54:21.898 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.916+0000 7f4bf63c1640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f4be8006e30 0x7f4be80cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:44181/0 (socket says 172.21.15.79:44181) 2024-03-29T10:54:21.898 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.916+0000 7f4bf63c1640 1 -- 172.21.15.79:0/1724531988 learned_addr learned my addr 172.21.15.79:0/1724531988 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:21.899 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.917+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227120361 0 0) 0x7f4be8035970 con 0x7f4be8006e30 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.917+0000 7f4bf53bf640 10 monclient(hunting): _negotiate 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.917+0000 7f4bf53bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.917+0000 7f4bf53bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.917+0000 7f4bf53bf640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.917+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc003520 con 0x7f4be8006e30 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.918+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600251348 0 0) 0x7f4bdc003520 con 0x7f4be8006e30 2024-03-29T10:54:21.900 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.918+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 >> v1:172.21.15.79:6790/0 conn(0x7f4be80c9670 legacy=0x7f4be80d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:21.906 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 >> v1:172.21.15.82:6789/0 conn(0x7f4be8035e00 legacy=0x7f4be80c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:21.906 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: _finish_hunting 0 2024-03-29T10:54:21.906 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 1 monclient: found mon.noname-a 2024-03-29T10:54:21.906 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:21.906 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be8034e20 con 0x7f4be8006e30 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: _finish_auth 0 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: _check_auth_tickets 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4bec001690 con 0x7f4be8006e30 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf53bf640 10 monclient: dump: 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:21.907 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:21.908 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:21.908 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:21.908 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:21.908 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf73c3640 5 monclient: authenticate success, global_id 4446 2024-03-29T10:54:21.908 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.921+0000 7f4bf73c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:21.910 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.926+0000 7f4bf53bf640 1 -- 172.21.15.79:0/1724531988 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bec001860 con 0x7f4be8006e30 2024-03-29T10:54:21.910 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.926+0000 7f4bf53bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:21.910 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.926+0000 7f4bf73c3640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:21.910 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.926+0000 7f4bf73c3640 4 set_mon_vals no callback set 2024-03-29T10:54:21.910 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.927+0000 7f4bf73c3640 10 monclient: shutdown 2024-03-29T10:54:21.910 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.927+0000 7f4bf73c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:21.911 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.927+0000 7f4bf73c3640 1 -- 172.21.15.79:0/1724531988 >> v1:172.21.15.79:6789/0 conn(0x7f4be8006e30 legacy=0x7f4be80cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:21.911 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.927+0000 7f4bf73c3640 1 -- 172.21.15.79:0/1724531988 shutdown_connections 2024-03-29T10:54:21.911 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.927+0000 7f4bf73c3640 1 -- 172.21.15.79:0/1724531988 >> 172.21.15.79:0/1724531988 conn(0x7f4be8005c00 msgr2=0x7f4be8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:21.913 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.929+0000 7f4bf73c3640 1 -- 172.21.15.79:0/1724531988 shutdown_connections 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.929+0000 7f4bf73c3640 1 -- 172.21.15.79:0/1724531988 wait complete. 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 10 monclient: build_initial_monmap 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 10 monclient: monmap: 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:21.914 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:21.915 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 1 librados: starting msgr at 2024-03-29T10:54:21.915 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 1 librados: starting objecter 2024-03-29T10:54:21.915 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 1 Processor -- start 2024-03-29T10:54:21.917 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 1 -- start start 2024-03-29T10:54:21.917 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 1 librados: setting wanted keys 2024-03-29T10:54:21.917 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 1 librados: calling monclient init 2024-03-29T10:54:21.917 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.930+0000 7f4bf73c3640 10 monclient: init 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient(hunting): picked mon.noname-a con 0x7f4be8035e00 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient(hunting): picked mon.noname-c con 0x7f4be80c9670 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient(hunting): picked mon.noname-b con 0x7f4be80ced20 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be80ce970 con 0x7f4be8035e00 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be8030030 con 0x7f4be80ced20 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be816a000 con 0x7f4be80c9670 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:21.918 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.931+0000 7f4bf73c3640 10 monclient(hunting): authenticate will time out at 1381.224928s 2024-03-29T10:54:21.921 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.938+0000 7f4bf6bc2640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f4be80ced20 0x7f4be81670a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:51605/0 (socket says 172.21.15.79:51605) 2024-03-29T10:54:21.921 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.938+0000 7f4bf6bc2640 1 -- 172.21.15.79:0/2349994239 learned_addr learned my addr 172.21.15.79:0/2349994239 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:21.921 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.938+0000 7f4be6ffd640 10 client.?.objecter ms_handle_connect 0x7f4be80ced20 2024-03-29T10:54:21.921 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.938+0000 7f4be6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:21.921 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132795479 0 0) 0x7f4be8030030 con 0x7f4be80ced20 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd4004eb0 con 0x7f4be80ced20 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 10 client.?.objecter ms_handle_connect 0x7f4be8035e00 2024-03-29T10:54:21.922 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.939+0000 7f4be6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 10 client.?.objecter ms_handle_connect 0x7f4be80c9670 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966268889 0 0) 0x7f4bd4004eb0 con 0x7f4be80ced20 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be8030030 con 0x7f4be80ced20 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4bf004fdd0 con 0x7f4be80ced20 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:21.923 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.941+0000 7f4be6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:21.924 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.942+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018326906 0 0) 0x7f4be816a000 con 0x7f4be80c9670 2024-03-29T10:54:21.925 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.942+0000 7f4be6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:21.925 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.942+0000 7f4be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:21.925 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.942+0000 7f4be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:21.925 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.942+0000 7f4be6ffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:54:21.925 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.942+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd4004eb0 con 0x7f4be80c9670 2024-03-29T10:54:21.928 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1703204987 0 0) 0x7f4be8030030 con 0x7f4be80ced20 2024-03-29T10:54:21.928 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 >> v1:172.21.15.79:6790/0 conn(0x7f4be80c9670 legacy=0x7f4be816a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 >> v1:172.21.15.79:6789/0 conn(0x7f4be8035e00 legacy=0x7f4be80cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 monclient: found mon.b 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be816abf0 con 0x7f4be80ced20 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:21.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 5 monclient: authenticate success, global_id 4438 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 10 monclient: _renew_subs 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be81677b0 con 0x7f4be80ced20 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 10 monclient: _renew_subs 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be802f590 con 0x7f4be80ced20 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf73c3640 1 librados: init done 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bf004f400 con 0x7f4be80ced20 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:21.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f4bf0075710 con 0x7f4be80ced20 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4be6ffd640 10 monclient: dump: 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:21.931 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.946+0000 7f4bf4bbe640 4 set_mon_vals no callback set 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.949+0000 7f4bf73c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.949+0000 7f4bf73c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.949+0000 7f4bf73c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:21.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.949+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd8005080 con 0x7f4be80ced20 2024-03-29T10:54:21.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.951+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f4bf0067bf0 con 0x7f4be80ced20 2024-03-29T10:54:21.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.952+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f4bf00ff590 con 0x7f4be80ced20 2024-03-29T10:54:21.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.952+0000 7f4be6ffd640 10 client.4438.objecter ms_dispatch 0x7f4bf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:21.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.952+0000 7f4be6ffd640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:21.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.952+0000 7f4be6ffd640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:21.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.952+0000 7f4be6ffd640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T10:54:21.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.953+0000 7f4bf63c1640 10 monclient: get_auth_request con 0x7f4bd406db60 auth_method 0 2024-03-29T10:54:21.935 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.953+0000 7f4be6ffd640 10 client.4438.objecter ms_handle_connect 0x7f4bd406db60 2024-03-29T10:54:21.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.959+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bf00cae70 con 0x7f4be80ced20 2024-03-29T10:54:21.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.959+0000 7f4be6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:21.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:21.959+0000 7f4be6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:22.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.317+0000 7f4bf73c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:22.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.317+0000 7f4bf73c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:22.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.317+0000 7f4bf73c3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:22.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.317+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bd8005330 con 0x7f4be80ced20 2024-03-29T10:54:22.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.319+0000 7f4be6ffd640 1 -- 172.21.15.79:0/2349994239 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f4bf00ccda0 con 0x7f4be80ced20 2024-03-29T10:54:22.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.319+0000 7f4be6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:22.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.319+0000 7f4be6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:22.302 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:22.302 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.323+0000 7f4bf73c3640 10 librados: watch_flush enter 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 10 librados: watch_flush exit 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 10 client.4438.objecter successfully canceled tick 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 >> v1:172.21.15.79:6816/4290694627 conn(0x7f4bd406db60 legacy=0x7f4bd4070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 10 monclient: shutdown 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:22.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.326+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 >> v1:172.21.15.82:6789/0 conn(0x7f4be80ced20 legacy=0x7f4be81670a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.327+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 shutdown_connections 2024-03-29T10:54:22.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.327+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 >> 172.21.15.79:0/2349994239 conn(0x7f4be8005c00 msgr2=0x7f4be80c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:22.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.330+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 shutdown_connections 2024-03-29T10:54:22.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.330+0000 7f4bf73c3640 1 -- 172.21.15.79:0/2349994239 wait complete. 2024-03-29T10:54:22.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:22.330+0000 7f4bf73c3640 1 librados: shutdown 2024-03-29T10:54:22.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:29.143 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:29.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.302+0000 7fb2ef343640 10 monclient: get_monmap_and_config 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.302+0000 7fb2ef343640 10 monclient: build_initial_monmap 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.302+0000 7fb2ef343640 10 monclient: monmap: 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:29.303047+0000 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:29.303047+0000 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:29.284 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:29.285 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:29.285 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:29.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.303+0000 7fb2ef343640 1 Processor -- start 2024-03-29T10:54:29.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.303+0000 7fb2ef343640 1 -- start start 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.303+0000 7fb2ef343640 10 monclient: init 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient(hunting): picked mon.noname-b con 0x7fb2e80b0770 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient(hunting): picked mon.noname-c con 0x7fb2e814a700 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ed0df640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb2e80b0770 0x7fb2e8149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:46448/0 (socket says 172.21.15.79:46448) 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ed0df640 1 -- 172.21.15.79:0/1104931439 learned_addr learned my addr 172.21.15.79:0/1104931439 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient(hunting): picked mon.noname-a con 0x7fb2e80ac5a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80aba70 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80aecb0 con 0x7fb2e814a700 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80af150 con 0x7fb2e80b0770 2024-03-29T10:54:29.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.304+0000 7fb2ef343640 10 monclient(hunting): authenticate will time out at 1388.597043s 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783460973 0 0) 0x7fb2e80aba70 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d4003520 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800255169 0 0) 0x7fb2e80af150 con 0x7fb2e80b0770 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:54:29.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e80aba70 con 0x7fb2e80b0770 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724516450 0 0) 0x7fb2d4003520 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 >> v1:172.21.15.79:6790/0 conn(0x7fb2e80b0770 legacy=0x7fb2e8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.305+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 >> v1:172.21.15.82:6789/0 conn(0x7fb2e814a700 legacy=0x7fb2e80abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 1 monclient: found mon.noname-a 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80ac9d0 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: _finish_auth 0 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb2d8001690 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:29.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.306+0000 7fb2dffff640 10 monclient: dump: 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2ef343640 5 monclient: authenticate success, global_id 4452 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2ef343640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2d80017f0 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2dffff640 1 -- 172.21.15.79:0/1104931439 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb2d8001c00 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.307+0000 7fb2dffff640 10 monclient: dump: 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:29.297 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.308+0000 7fb2ef343640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.308+0000 7fb2ef343640 4 set_mon_vals no callback set 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.308+0000 7fb2ef343640 10 monclient: shutdown 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.308+0000 7fb2ef343640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.308+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 >> v1:172.21.15.79:6789/0 conn(0x7fb2e80ac5a0 legacy=0x7fb2e8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.309+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 shutdown_connections 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.309+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 >> 172.21.15.79:0/1104931439 conn(0x7fb2e806b660 msgr2=0x7fb2e806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.309+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 shutdown_connections 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.309+0000 7fb2ef343640 1 -- 172.21.15.79:0/1104931439 wait complete. 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 10 monclient: build_initial_monmap 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 10 monclient: monmap: 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:29.298 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 1 librados: starting msgr at 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 1 librados: starting objecter 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 1 Processor -- start 2024-03-29T10:54:29.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 1 -- start start 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 1 librados: setting wanted keys 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 1 librados: calling monclient init 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.310+0000 7fb2ef343640 10 monclient: init 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient(hunting): picked mon.noname-b con 0x7fb2e80ac5a0 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient(hunting): picked mon.noname-c con 0x7fb2e80b0770 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient(hunting): picked mon.noname-a con 0x7fb2e814a700 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e8128510 con 0x7fb2e814a700 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e81459a0 con 0x7fb2e80ac5a0 2024-03-29T10:54:29.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e8145c50 con 0x7fb2e80b0770 2024-03-29T10:54:29.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:29.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ef343640 10 monclient(hunting): authenticate will time out at 1388.603848s 2024-03-29T10:54:29.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ec8de640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb2e80b0770 0x7fb2e8144b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47462/0 (socket says 172.21.15.79:47462) 2024-03-29T10:54:29.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.311+0000 7fb2ec8de640 1 -- 172.21.15.79:0/2461333045 learned_addr learned my addr 172.21.15.79:0/2461333045 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:29.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 client.?.objecter ms_handle_connect 0x7fb2e814a700 2024-03-29T10:54:29.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 client.?.objecter ms_handle_connect 0x7fb2e80b0770 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396086492 0 0) 0x7fb2e8128510 con 0x7fb2e814a700 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2cc004eb0 con 0x7fb2e814a700 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587659373 0 0) 0x7fb2e8145c50 con 0x7fb2e80b0770 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:29.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e8128510 con 0x7fb2e80b0770 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2195298498 0 0) 0x7fb2cc004eb0 con 0x7fb2e814a700 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2e8145c50 con 0x7fb2e814a700 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb2d8001b00 con 0x7fb2e814a700 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): dump: 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.309 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1204344653 0 0) 0x7fb2e8145c50 con 0x7fb2e814a700 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 >> v1:172.21.15.79:6790/0 conn(0x7fb2e80b0770 legacy=0x7fb2e8144b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.312+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 >> v1:172.21.15.82:6789/0 conn(0x7fb2e80ac5a0 legacy=0x7fb2e81267b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:29.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 1 monclient: found mon.a 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8126290 con 0x7fb2e814a700 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 5 monclient: authenticate success, global_id 4455 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 10 monclient: _renew_subs 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e8125e40 con 0x7fb2e814a700 2024-03-29T10:54:29.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 10 monclient: _renew_subs 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e81dc350 con 0x7fb2e814a700 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.313+0000 7fb2ef343640 1 librados: init done 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2d80038a0 con 0x7fb2e814a700 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb2d8001cc0 con 0x7fb2e814a700 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:29.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2ddffb640 10 monclient: dump: 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.315+0000 7fb2deffd640 4 set_mon_vals no callback set 2024-03-29T10:54:29.313 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.316+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fb2d8019a20 con 0x7fb2e814a700 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fb2d80838b0 con 0x7fb2e814a700 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ddffb640 10 client.4455.objecter ms_dispatch 0x7fb2e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ddffb640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ddffb640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ed0df640 10 monclient: get_auth_request con 0x7fb2cc06dcb0 auth_method 0 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ddffb640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ddffb640 10 client.4455.objecter ms_handle_connect 0x7fb2cc06dcb0 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ef343640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ef343640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ef343640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:29.314 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.317+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c8005080 con 0x7fb2e814a700 2024-03-29T10:54:29.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.326+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2d804f190 con 0x7fb2e814a700 2024-03-29T10:54:29.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.326+0000 7fb2ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:29.315 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.326+0000 7fb2ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:29.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.637+0000 7fb2ef343640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:29.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.637+0000 7fb2ef343640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:29.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.637+0000 7fb2ef343640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:29.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.637+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2c8005330 con 0x7fb2e814a700 2024-03-29T10:54:29.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.638+0000 7fb2ddffb640 1 -- 172.21.15.79:0/2461333045 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fb2d80510c0 con 0x7fb2e814a700 2024-03-29T10:54:29.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.638+0000 7fb2ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:29.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.638+0000 7fb2ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:29.620 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:29.620 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:29.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.642+0000 7fb2ef343640 10 librados: watch_flush enter 2024-03-29T10:54:29.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.642+0000 7fb2ef343640 10 librados: watch_flush exit 2024-03-29T10:54:29.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.642+0000 7fb2ef343640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.642+0000 7fb2ef343640 10 client.4455.objecter successfully canceled tick 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 >> v1:172.21.15.79:6816/4290694627 conn(0x7fb2cc06dcb0 legacy=0x7fb2cc070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 10 monclient: shutdown 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 >> v1:172.21.15.79:6789/0 conn(0x7fb2e814a700 legacy=0x7fb2e8145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 shutdown_connections 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 >> 172.21.15.79:0/2461333045 conn(0x7fb2e806b660 msgr2=0x7fb2e8071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 shutdown_connections 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 -- 172.21.15.79:0/2461333045 wait complete. 2024-03-29T10:54:29.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:29.643+0000 7fb2ef343640 1 librados: shutdown 2024-03-29T10:54:29.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:36.444 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:36.587 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.606+0000 7f27cd625640 10 monclient: get_monmap_and_config 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.606+0000 7f27cd625640 10 monclient: build_initial_monmap 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.606+0000 7f27cd625640 10 monclient: monmap: 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:36.607167+0000 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:36.607167+0000 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:36.588 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:36.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.608+0000 7f27cd625640 1 Processor -- start 2024-03-29T10:54:36.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.608+0000 7f27cd625640 1 -- start start 2024-03-29T10:54:36.589 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.608+0000 7f27cd625640 10 monclient: init 2024-03-29T10:54:36.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:36.593 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient(hunting): picked mon.noname-c con 0x7f27c80b0860 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient(hunting): picked mon.noname-a con 0x7f27c813ef30 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient(hunting): picked mon.noname-b con 0x7f27c80af8e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80b0c40 con 0x7f27c813ef30 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80aed10 con 0x7f27c80b0860 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80afd50 con 0x7f27c80af8e0 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.609+0000 7f27cd625640 10 monclient(hunting): authenticate will time out at 1395.901437s 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.611+0000 7f27c67fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f27c813ef30 0x7f27c80af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:50048/0 (socket says 172.21.15.79:50048) 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.611+0000 7f27c67fc640 1 -- 172.21.15.79:0/3440098822 learned_addr learned my addr 172.21.15.79:0/3440098822 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769895923 0 0) 0x7f27c80b0c40 con 0x7f27c813ef30 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:36.594 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b0003520 con 0x7f27c813ef30 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937712608 0 0) 0x7f27c80afd50 con 0x7f27c80af8e0 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c80b0c40 con 0x7f27c80af8e0 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166579980 0 0) 0x7f27b0003520 con 0x7f27c813ef30 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 >> v1:172.21.15.79:6790/0 conn(0x7f27c80af8e0 legacy=0x7f27c8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:36.595 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.612+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 >> v1:172.21.15.82:6789/0 conn(0x7f27c80b0860 legacy=0x7f27c813e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:36.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:36.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 1 monclient: found mon.noname-a 2024-03-29T10:54:36.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:36.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:54:36.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80ab1b0 con 0x7f27c813ef30 2024-03-29T10:54:36.596 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.614+0000 7f27c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27cd625640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27cd625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f27c0001a10 con 0x7f27c813ef30 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: dump: 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:36.597 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27cd625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c0001c30 con 0x7f27c813ef30 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27cd625640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27cd625640 4 set_mon_vals no callback set 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 1 -- 172.21.15.79:0/3440098822 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f27c0001fc0 con 0x7f27c813ef30 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.615+0000 7f27c5ffb640 10 monclient: dump: 2024-03-29T10:54:36.598 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:36.599 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:36.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 10 monclient: shutdown 2024-03-29T10:54:36.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:36.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 1 -- 172.21.15.79:0/3440098822 >> v1:172.21.15.79:6789/0 conn(0x7f27c813ef30 legacy=0x7f27c80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:36.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 1 -- 172.21.15.79:0/3440098822 shutdown_connections 2024-03-29T10:54:36.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 1 -- 172.21.15.79:0/3440098822 >> 172.21.15.79:0/3440098822 conn(0x7f27c806b660 msgr2=0x7f27c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:36.600 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 1 -- 172.21.15.79:0/3440098822 shutdown_connections 2024-03-29T10:54:36.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.616+0000 7f27cd625640 1 -- 172.21.15.79:0/3440098822 wait complete. 2024-03-29T10:54:36.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.617+0000 7f27cd625640 10 monclient: build_initial_monmap 2024-03-29T10:54:36.601 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.617+0000 7f27cd625640 10 monclient: monmap: 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:36.602 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 1 librados: starting msgr at 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 1 librados: starting objecter 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 1 Processor -- start 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 1 -- start start 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 1 librados: setting wanted keys 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 1 librados: calling monclient init 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.620+0000 7f27cd625640 10 monclient: init 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient(hunting): picked mon.noname-c con 0x7f27c80af8e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient(hunting): picked mon.noname-b con 0x7f27c80b0860 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:36.604 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient(hunting): picked mon.noname-a con 0x7f27c813ef30 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c81365a0 con 0x7f27c813ef30 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c8135a70 con 0x7f27c80b0860 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c80b2220 con 0x7f27c80af8e0 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.621+0000 7f27cd625640 10 monclient(hunting): authenticate will time out at 1395.913654s 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.624+0000 7f27c77fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f27c813ef30 0x7f27c81e6310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:53160/0 (socket says 172.21.15.79:53160) 2024-03-29T10:54:36.605 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.624+0000 7f27c77fe640 1 -- 172.21.15.79:0/1530727202 learned_addr learned my addr 172.21.15.79:0/1530727202 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:36.608 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 10 client.?.objecter ms_handle_connect 0x7f27c813ef30 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201707382 0 0) 0x7f27c81365a0 con 0x7f27c813ef30 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b4004eb0 con 0x7f27c813ef30 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.625+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3726112975 0 0) 0x7f27b4004eb0 con 0x7f27c813ef30 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27c81365a0 con 0x7f27c813ef30 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f27bc001990 con 0x7f27c813ef30 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): dump: 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:36.609 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 client.?.objecter ms_handle_connect 0x7f27c80af8e0 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711303198 0 0) 0x7f27c80b2220 con 0x7f27c80af8e0 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:36.610 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.626+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b4004eb0 con 0x7f27c80af8e0 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336514302 0 0) 0x7f27b4004eb0 con 0x7f27c80af8e0 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27c80b2220 con 0x7f27c80af8e0 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 66011092 0 0) 0x7f27c81365a0 con 0x7f27c813ef30 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 >> v1:172.21.15.79:6790/0 conn(0x7f27c80af8e0 legacy=0x7f27c8135e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 >> v1:172.21.15.82:6789/0 conn(0x7f27c80b0860 legacy=0x7f27c80b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:36.611 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 monclient: found mon.a 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80b36b0 con 0x7f27c813ef30 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 5 monclient: authenticate success, global_id 4461 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 10 monclient: _renew_subs 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c81c9040 con 0x7f27c813ef30 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27bc001c90 con 0x7f27c813ef30 2024-03-29T10:54:36.613 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27c57fa640 4 set_mon_vals no callback set 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 10 monclient: _renew_subs 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c81e7a30 con 0x7f27c813ef30 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.627+0000 7f27cd625640 1 librados: init done 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.628+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f27bc003a60 con 0x7f27c813ef30 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.628+0000 7f27af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.628+0000 7f27af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.628+0000 7f27af7fe640 10 monclient: dump: 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:36.614 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.629+0000 7f27cd625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.629+0000 7f27cd625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.629+0000 7f27cd625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.629+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c8138430 con 0x7f27c813ef30 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.629+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f27bc003c40 con 0x7f27c813ef30 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.630+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f27bc054720 con 0x7f27c813ef30 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.630+0000 7f27af7fe640 10 client.4461.objecter ms_dispatch 0x7f27c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:36.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.630+0000 7f27af7fe640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:36.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.630+0000 7f27af7fe640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:36.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.630+0000 7f27af7fe640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T10:54:36.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.632+0000 7f27c6ffd640 10 monclient: get_auth_request con 0x7f27b406ded0 auth_method 0 2024-03-29T10:54:36.616 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.633+0000 7f27af7fe640 10 client.4461.objecter ms_handle_connect 0x7f27b406ded0 2024-03-29T10:54:36.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.641+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27bc0509c0 con 0x7f27c813ef30 2024-03-29T10:54:36.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.642+0000 7f27af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:36.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.642+0000 7f27af7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:36.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.969+0000 7f27cd625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:36.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.969+0000 7f27cd625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:36.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.969+0000 7f27cd625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:36.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.969+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27c81c91d0 con 0x7f27c813ef30 2024-03-29T10:54:36.952 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:36.953 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:36.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.971+0000 7f27af7fe640 1 -- 172.21.15.79:0/1530727202 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f27bc0509c0 con 0x7f27c813ef30 2024-03-29T10:54:36.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.971+0000 7f27af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:36.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.971+0000 7f27af7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:36.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.974+0000 7f27cd625640 10 librados: watch_flush enter 2024-03-29T10:54:36.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.975+0000 7f27cd625640 10 librados: watch_flush exit 2024-03-29T10:54:36.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.975+0000 7f27cd625640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T10:54:36.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.975+0000 7f27cd625640 10 client.4461.objecter successfully canceled tick 2024-03-29T10:54:36.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 >> v1:172.21.15.79:6816/4290694627 conn(0x7f27b406ded0 legacy=0x7f27b4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:36.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 10 monclient: shutdown 2024-03-29T10:54:36.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:36.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 >> v1:172.21.15.79:6789/0 conn(0x7f27c813ef30 legacy=0x7f27c81e6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:36.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 shutdown_connections 2024-03-29T10:54:36.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 >> 172.21.15.79:0/1530727202 conn(0x7f27c806b660 msgr2=0x7f27c813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:36.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 shutdown_connections 2024-03-29T10:54:36.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 -- 172.21.15.79:0/1530727202 wait complete. 2024-03-29T10:54:36.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:36.978+0000 7f27cd625640 1 librados: shutdown 2024-03-29T10:54:36.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:43.784 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:43.930 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.948+0000 7f0e7a91c640 10 monclient: get_monmap_and_config 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.948+0000 7f0e7a91c640 10 monclient: build_initial_monmap 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.948+0000 7f0e7a91c640 10 monclient: monmap: 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:43.949219+0000 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:43.949219+0000 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:43.931 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.949+0000 7f0e7a91c640 1 Processor -- start 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.949+0000 7f0e7a91c640 1 -- start start 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.949+0000 7f0e7a91c640 10 monclient: init 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient(hunting): picked mon.noname-a con 0x7f0e740a9930 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient(hunting): picked mon.noname-b con 0x7f0e74149dc0 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient(hunting): picked mon.noname-c con 0x7f0e7414ff40 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:43.932 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740aae60 con 0x7f0e740a9930 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740b4310 con 0x7f0e7414ff40 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740afdd0 con 0x7f0e74149dc0 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.950+0000 7f0e7a91c640 10 monclient(hunting): authenticate will time out at 1403.242802s 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e737fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f0e74149dc0 0x7f0e7414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47184/0 (socket says 172.21.15.79:47184) 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e737fe640 1 -- 172.21.15.79:0/230985375 learned_addr learned my addr 172.21.15.79:0/230985375 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:43.933 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814199780 0 0) 0x7f0e740afdd0 con 0x7f0e74149dc0 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e72ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e72ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.951+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c003520 con 0x7f0e74149dc0 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.952+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664502987 0 0) 0x7f0e5c003520 con 0x7f0e74149dc0 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.952+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 >> v1:172.21.15.82:6789/0 conn(0x7f0e7414ff40 legacy=0x7f0e74152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:43.934 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.954+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 >> v1:172.21.15.79:6789/0 conn(0x7f0e740a9930 legacy=0x7f0e74143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 1 monclient: found mon.noname-b 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7414a950 con 0x7f0e74149dc0 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0e6c001a10 con 0x7f0e74149dc0 2024-03-29T10:54:43.937 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:43.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:43.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: dump: 2024-03-29T10:54:43.939 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:43.939 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:43.939 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e7a91c640 5 monclient: authenticate success, global_id 4535 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e7a91c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e6c001c70 con 0x7f0e74149dc0 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 1 -- 172.21.15.79:0/230985375 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0e6c0020c0 con 0x7f0e74149dc0 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:43.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e72ffd640 10 monclient: dump: 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e7a91c640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.955+0000 7f0e7a91c640 4 set_mon_vals no callback set 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.956+0000 7f0e7a91c640 10 monclient: shutdown 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.956+0000 7f0e7a91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:43.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.956+0000 7f0e7a91c640 1 -- 172.21.15.79:0/230985375 >> v1:172.21.15.79:6790/0 conn(0x7f0e74149dc0 legacy=0x7f0e7414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:43.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.956+0000 7f0e7a91c640 1 -- 172.21.15.79:0/230985375 shutdown_connections 2024-03-29T10:54:43.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.956+0000 7f0e7a91c640 1 -- 172.21.15.79:0/230985375 >> 172.21.15.79:0/230985375 conn(0x7f0e740b35e0 msgr2=0x7f0e740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:43.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.959+0000 7f0e7a91c640 1 -- 172.21.15.79:0/230985375 shutdown_connections 2024-03-29T10:54:43.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.960+0000 7f0e7a91c640 1 -- 172.21.15.79:0/230985375 wait complete. 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 10 monclient: build_initial_monmap 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 10 monclient: monmap: 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 1 librados: starting msgr at 2024-03-29T10:54:43.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 1 librados: starting objecter 2024-03-29T10:54:43.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 1 Processor -- start 2024-03-29T10:54:43.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 1 -- start start 2024-03-29T10:54:43.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 1 librados: setting wanted keys 2024-03-29T10:54:43.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 1 librados: calling monclient init 2024-03-29T10:54:43.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.961+0000 7f0e7a91c640 10 monclient: init 2024-03-29T10:54:43.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient(hunting): picked mon.noname-c con 0x7f0e740a9930 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7414ff40 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient(hunting): picked mon.noname-a con 0x7f0e740722b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74071130 con 0x7f0e740722b0 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7406fb60 con 0x7f0e7414ff40 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7406fcf0 con 0x7f0e740a9930 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:43.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.962+0000 7f0e7a91c640 10 monclient(hunting): authenticate will time out at 1403.254720s 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e73fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f0e740a9930 0x7f0e7406f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:48786/0 (socket says 172.21.15.79:48786) 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e73fff640 1 -- 172.21.15.79:0/2026387590 learned_addr learned my addr 172.21.15.79:0/2026387590 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 10 client.?.objecter ms_handle_connect 0x7f0e740a9930 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144801789 0 0) 0x7f0e7406fcf0 con 0x7f0e740a9930 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:43.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.963+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60004eb0 con 0x7f0e740a9930 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091818510 0 0) 0x7f0e60004eb0 con 0x7f0e740a9930 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e60005300 con 0x7f0e740a9930 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0e68001920 con 0x7f0e740a9930 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:43.949 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:43.950 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:43.950 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:43.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:43.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1171607684 0 0) 0x7f0e60005300 con 0x7f0e740a9930 2024-03-29T10:54:43.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.964+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 >> v1:172.21.15.82:6789/0 conn(0x7f0e7414ff40 legacy=0x7f0e74071ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:43.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.965+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 >> v1:172.21.15.79:6789/0 conn(0x7f0e740722b0 legacy=0x7f0e741e6190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 1 monclient: found mon.c 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e741ea8e0 con 0x7f0e740a9930 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 10 monclient: _finish_auth 0 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.967+0000 7f0e70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:54:43.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 10 monclient: _renew_subs 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e741e78b0 con 0x7f0e740a9930 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 10 monclient: _renew_subs 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e741e7de0 con 0x7f0e740a9930 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.968+0000 7f0e7a91c640 1 librados: init done 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.969+0000 7f0e7a91c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.969+0000 7f0e7a91c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.969+0000 7f0e7a91c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:54:43.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.969+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e740726c0 con 0x7f0e740a9930 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e68001c40 con 0x7f0e740a9930 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0e68001920 con 0x7f0e740a9930 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e70ff9640 10 monclient: dump: 2024-03-29T10:54:43.955 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:43.956 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f0e680038a0 con 0x7f0e740a9930 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.974+0000 7f0e71ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f0e68084710 con 0x7f0e740a9930 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e70ff9640 10 client.4538.objecter ms_dispatch 0x7f0e74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e70ff9640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e70ff9640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e737fe640 10 monclient: get_auth_request con 0x7f0e6006dfb0 auth_method 0 2024-03-29T10:54:43.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.975+0000 7f0e70ff9640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T10:54:43.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.976+0000 7f0e70ff9640 10 client.4538.objecter ms_handle_connect 0x7f0e6006dfb0 2024-03-29T10:54:43.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.984+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e6804fff0 con 0x7f0e740a9930 2024-03-29T10:54:43.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.984+0000 7f0e70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:43.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:43.984+0000 7f0e70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:44.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.307+0000 7f0e7a91c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:44.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.307+0000 7f0e7a91c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:44.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.307+0000 7f0e7a91c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:54:44.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.307+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e74143f10 con 0x7f0e740a9930 2024-03-29T10:54:44.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.308+0000 7f0e70ff9640 1 -- 172.21.15.79:0/2026387590 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f0e6804fff0 con 0x7f0e740a9930 2024-03-29T10:54:44.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.308+0000 7f0e70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:44.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.308+0000 7f0e70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:44.292 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:44.292 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 10 librados: watch_flush enter 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 10 librados: watch_flush exit 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 10 client.4538.objecter successfully canceled tick 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 >> v1:172.21.15.79:6816/4290694627 conn(0x7f0e6006dfb0 legacy=0x7f0e60070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 10 monclient: shutdown 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 >> v1:172.21.15.79:6790/0 conn(0x7f0e740a9930 legacy=0x7f0e7406f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 shutdown_connections 2024-03-29T10:54:44.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.313+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 >> 172.21.15.79:0/2026387590 conn(0x7f0e740b35e0 msgr2=0x7f0e74143800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:44.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.314+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 shutdown_connections 2024-03-29T10:54:44.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.314+0000 7f0e7a91c640 1 -- 172.21.15.79:0/2026387590 wait complete. 2024-03-29T10:54:44.296 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:44.314+0000 7f0e7a91c640 1 librados: shutdown 2024-03-29T10:54:44.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:51.120 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:51.271 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.289+0000 7f317bd89640 10 monclient: get_monmap_and_config 2024-03-29T10:54:51.271 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.289+0000 7f317bd89640 10 monclient: build_initial_monmap 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.290+0000 7f317bd89640 10 monclient: monmap: 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:51.290914+0000 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:51.290914+0000 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:51.273 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.292+0000 7f317bd89640 1 Processor -- start 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.292+0000 7f317bd89640 1 -- start start 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.292+0000 7f317bd89640 10 monclient: init 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient(hunting): picked mon.noname-b con 0x7f31740a9750 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient(hunting): picked mon.noname-a con 0x7f31740aacf0 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient(hunting): picked mon.noname-c con 0x7f31740af220 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31740ac2f0 con 0x7f31740aacf0 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31740acaf0 con 0x7f31740af220 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31740ac710 con 0x7f31740a9750 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:51.275 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.293+0000 7f317bd89640 10 monclient(hunting): authenticate will time out at 1410.585495s 2024-03-29T10:54:51.278 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.296+0000 7f3179324640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f31740aacf0 0x7f317413e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:45341/0 (socket says 172.21.15.79:45341) 2024-03-29T10:54:51.278 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.296+0000 7f3179324640 1 -- 172.21.15.79:0/4156267828 learned_addr learned my addr 172.21.15.79:0/4156267828 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.297+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588284145 0 0) 0x7f31740ac2f0 con 0x7f31740aacf0 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.297+0000 7f3178b23640 10 monclient(hunting): _negotiate 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.297+0000 7f3178b23640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.297+0000 7f3178b23640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.297+0000 7f3178b23640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.297+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3160003520 con 0x7f31740aacf0 2024-03-29T10:54:51.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.298+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 341943163 0 0) 0x7f3160003520 con 0x7f31740aacf0 2024-03-29T10:54:51.280 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.298+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 >> v1:172.21.15.79:6790/0 conn(0x7f31740a9750 legacy=0x7f31740ad010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:51.281 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.299+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 >> v1:172.21.15.82:6789/0 conn(0x7f31740af220 legacy=0x7f3174149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.299+0000 7f3178b23640 10 monclient: _finish_hunting 0 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.299+0000 7f3178b23640 1 monclient: found mon.noname-a 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.299+0000 7f3178b23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.299+0000 7f3178b23640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.299+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31740ac8a0 con 0x7f31740aacf0 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 10 monclient: _finish_auth 0 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 10 monclient: _check_auth_tickets 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f317bd89640 5 monclient: authenticate success, global_id 4464 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f317bd89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f316c001a10 con 0x7f31740aacf0 2024-03-29T10:54:51.282 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 10 monclient: dump: 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:51.285 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f317bd89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 1 -- 172.21.15.79:0/4156267828 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f316c001ca0 con 0x7f31740aacf0 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f3178b23640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f317bd89640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.300+0000 7f317bd89640 4 set_mon_vals no callback set 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.301+0000 7f317bd89640 10 monclient: shutdown 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.301+0000 7f317bd89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.301+0000 7f317bd89640 1 -- 172.21.15.79:0/4156267828 >> v1:172.21.15.79:6789/0 conn(0x7f31740aacf0 legacy=0x7f317413e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.301+0000 7f317bd89640 1 -- 172.21.15.79:0/4156267828 shutdown_connections 2024-03-29T10:54:51.286 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.301+0000 7f317bd89640 1 -- 172.21.15.79:0/4156267828 >> 172.21.15.79:0/4156267828 conn(0x7f317406b660 msgr2=0x7f317406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.302+0000 7f317bd89640 1 -- 172.21.15.79:0/4156267828 shutdown_connections 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.302+0000 7f317bd89640 1 -- 172.21.15.79:0/4156267828 wait complete. 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 10 monclient: build_initial_monmap 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 10 monclient: monmap: 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 1 librados: starting msgr at 2024-03-29T10:54:51.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 1 librados: starting objecter 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 1 Processor -- start 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 1 -- start start 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 1 librados: setting wanted keys 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 1 librados: calling monclient init 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.303+0000 7f317bd89640 10 monclient: init 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient(hunting): picked mon.noname-c con 0x7f31740a9750 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient(hunting): picked mon.noname-a con 0x7f31740aacf0 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient(hunting): picked mon.noname-b con 0x7f31740af220 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3174130ad0 con 0x7f31740aacf0 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3174130d20 con 0x7f31740af220 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3174131000 con 0x7f31740a9750 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:51.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.304+0000 7f317bd89640 10 monclient(hunting): authenticate will time out at 1410.596823s 2024-03-29T10:54:51.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.309+0000 7f317a326640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f31740af220 0x7f31741e1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:41693/0 (socket says 172.21.15.79:41693) 2024-03-29T10:54:51.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.309+0000 7f317a326640 1 -- 172.21.15.79:0/2229034295 learned_addr learned my addr 172.21.15.79:0/2229034295 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:51.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.311+0000 7f316a7fc640 10 client.?.objecter ms_handle_connect 0x7f31740af220 2024-03-29T10:54:51.292 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.311+0000 7f316a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:51.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.312+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164358747 0 0) 0x7f3174130d20 con 0x7f31740af220 2024-03-29T10:54:51.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.312+0000 7f316a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:51.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.312+0000 7f316a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:51.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.312+0000 7f316a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:51.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.312+0000 7f316a7fc640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:54:51.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.312+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158004eb0 con 0x7f31740af220 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4275119378 0 0) 0x7f3158004eb0 con 0x7f31740af220 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3174130d20 con 0x7f31740af220 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 client.?.objecter ms_handle_connect 0x7f31740aacf0 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f3164003f60 con 0x7f31740af220 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 monclient(hunting): dump: 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:51.298 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.313+0000 7f316a7fc640 10 client.?.objecter ms_handle_connect 0x7f31740a9750 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2112494286 0 0) 0x7f3174130d20 con 0x7f31740af220 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 >> v1:172.21.15.79:6790/0 conn(0x7f31740a9750 legacy=0x7f317412f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 >> v1:172.21.15.79:6789/0 conn(0x7f31740aacf0 legacy=0x7f31740702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 1 monclient: found mon.b 2024-03-29T10:54:51.299 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317412fd00 con 0x7f31740af220 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f316a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 10 monclient: _renew_subs 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3174131420 con 0x7f31740af220 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 10 monclient: _renew_subs 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f317412eb10 con 0x7f31740af220 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.314+0000 7f317bd89640 1 librados: init done 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.315+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3164004130 con 0x7f31740af220 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.315+0000 7f316a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:51.300 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.315+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f31640010f0 con 0x7f31740af220 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.315+0000 7f316a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.315+0000 7f316a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.315+0000 7f316a7fc640 10 monclient: dump: 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:51.301 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.316+0000 7f316b7fe640 4 set_mon_vals no callback set 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f3164001250 con 0x7f31740af220 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f31640849d0 con 0x7f31740af220 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f316a7fc640 10 client.4441.objecter ms_dispatch 0x7f3174000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f316a7fc640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f316a7fc640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f316a7fc640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.317+0000 7f3179b25640 10 monclient: get_auth_request con 0x7f315806dbc0 auth_method 0 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.318+0000 7f316a7fc640 10 client.4441.objecter ms_handle_connect 0x7f315806dbc0 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.318+0000 7f317bd89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.318+0000 7f317bd89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.318+0000 7f317bd89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:51.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.318+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c005080 con 0x7f31740af220 2024-03-29T10:54:51.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.327+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31640500e0 con 0x7f31740af220 2024-03-29T10:54:51.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.327+0000 7f316a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:51.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.327+0000 7f316a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:51.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.651+0000 7f317bd89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:51.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.651+0000 7f317bd89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:51.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.651+0000 7f317bd89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:54:51.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.651+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f315c005330 con 0x7f31740af220 2024-03-29T10:54:51.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.653+0000 7f316a7fc640 1 -- 172.21.15.79:0/2229034295 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f31640521e0 con 0x7f31740af220 2024-03-29T10:54:51.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.653+0000 7f316a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:51.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.653+0000 7f316a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:51.635 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:51.635 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:51.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.657+0000 7f317bd89640 10 librados: watch_flush enter 2024-03-29T10:54:51.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.657+0000 7f317bd89640 10 librados: watch_flush exit 2024-03-29T10:54:51.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.657+0000 7f317bd89640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T10:54:51.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.657+0000 7f317bd89640 10 client.4441.objecter successfully canceled tick 2024-03-29T10:54:51.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 >> v1:172.21.15.79:6816/4290694627 conn(0x7f315806dbc0 legacy=0x7f3158070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:51.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 10 monclient: shutdown 2024-03-29T10:54:51.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:51.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 >> v1:172.21.15.82:6789/0 conn(0x7f31740af220 legacy=0x7f31741e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:51.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 shutdown_connections 2024-03-29T10:54:51.640 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 >> 172.21.15.79:0/2229034295 conn(0x7f317406b660 msgr2=0x7f31740b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:51.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 shutdown_connections 2024-03-29T10:54:51.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.659+0000 7f317bd89640 1 -- 172.21.15.79:0/2229034295 wait complete. 2024-03-29T10:54:51.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:51.660+0000 7f317bd89640 1 librados: shutdown 2024-03-29T10:54:51.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:58.466 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.634+0000 7f03fe83e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.634+0000 7f03fe83e640 10 monclient: build_initial_monmap 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.634+0000 7f03fe83e640 10 monclient: monmap: 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:54:58.635138+0000 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:54:58.635138+0000 2024-03-29T10:54:58.615 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:58.616 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.616 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:58.616 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:54:58.616 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:54:58.616 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.635+0000 7f03fe83e640 1 Processor -- start 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.635+0000 7f03fe83e640 1 -- start start 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.635+0000 7f03fe83e640 10 monclient: init 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient(hunting): picked mon.noname-b con 0x7f03f80b1f60 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:58.618 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient(hunting): picked mon.noname-c con 0x7f03f8071eb0 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient(hunting): picked mon.noname-a con 0x7f03f8143d10 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f806ef80 con 0x7f03f8143d10 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f806f260 con 0x7f03f8071eb0 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80b40c0 con 0x7f03f80b1f60 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.636+0000 7f03fe83e640 10 monclient(hunting): authenticate will time out at 1417.928913s 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f7fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f03f80b1f60 0x7f03f8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:55572/0 (socket says 172.21.15.79:55572) 2024-03-29T10:54:58.619 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f7fff640 1 -- 172.21.15.79:0/1229050476 learned_addr learned my addr 172.21.15.79:0/1229050476 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785195741 0 0) 0x7f03f80b40c0 con 0x7f03f80b1f60 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.637+0000 7f03f6ffd640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.638+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e0003520 con 0x7f03f80b1f60 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.638+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1931697703 0 0) 0x7f03e0003520 con 0x7f03f80b1f60 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.638+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 >> v1:172.21.15.82:6789/0 conn(0x7f03f8071eb0 legacy=0x7f03f8146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:58.620 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.639+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 >> v1:172.21.15.79:6789/0 conn(0x7f03f8143d10 legacy=0x7f03f8146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f8155d70 con 0x7f03f80b1f60 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03fe83e640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03fe83e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:58.624 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f03e8001690 con 0x7f03f80b1f60 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03f6ffd640 10 monclient: dump: 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.625 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.641+0000 7f03fe83e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.643+0000 7f03f6ffd640 1 -- 172.21.15.79:0/1229050476 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03e8001860 con 0x7f03f80b1f60 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.643+0000 7f03f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.643+0000 7f03fe83e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.643+0000 7f03fe83e640 4 set_mon_vals no callback set 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.644+0000 7f03fe83e640 10 monclient: shutdown 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.644+0000 7f03fe83e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.644+0000 7f03fe83e640 1 -- 172.21.15.79:0/1229050476 >> v1:172.21.15.79:6790/0 conn(0x7f03f80b1f60 legacy=0x7f03f8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.644+0000 7f03fe83e640 1 -- 172.21.15.79:0/1229050476 shutdown_connections 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.644+0000 7f03fe83e640 1 -- 172.21.15.79:0/1229050476 >> 172.21.15.79:0/1229050476 conn(0x7f03f8066a00 msgr2=0x7f03f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:58.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.646+0000 7f03fe83e640 1 -- 172.21.15.79:0/1229050476 shutdown_connections 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.646+0000 7f03fe83e640 1 -- 172.21.15.79:0/1229050476 wait complete. 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.647+0000 7f03fe83e640 10 monclient: build_initial_monmap 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.647+0000 7f03fe83e640 10 monclient: monmap: 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:54:58.629 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.647+0000 7f03fe83e640 1 librados: starting msgr at 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.647+0000 7f03fe83e640 1 librados: starting objecter 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.647+0000 7f03fe83e640 1 Processor -- start 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 1 -- start start 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 1 librados: setting wanted keys 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 1 librados: calling monclient init 2024-03-29T10:54:58.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient: init 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient(hunting): picked mon.noname-c con 0x7f03f80b1f60 addr v1:172.21.15.79:6790/0 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient(hunting): picked mon.noname-b con 0x7f03f8143d10 addr v1:172.21.15.82:6789/0 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient(hunting): picked mon.noname-a con 0x7f03f8138a70 addr v1:172.21.15.79:6789/0 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f81385c0 con 0x7f03f8138a70 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f8132920 con 0x7f03f8143d10 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80ac640 con 0x7f03f80b1f60 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.648+0000 7f03fe83e640 10 monclient(hunting): authenticate will time out at 1417.940983s 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.649+0000 7f03f7fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f03f80b1f60 0x7f03f80a9fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:59984/0 (socket says 172.21.15.79:59984) 2024-03-29T10:54:58.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.649+0000 7f03f7fff640 1 -- 172.21.15.79:0/3484725369 learned_addr learned my addr 172.21.15.79:0/3484725369 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:54:58.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 10 client.?.objecter ms_handle_connect 0x7f03f80b1f60 2024-03-29T10:54:58.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:58.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276527626 0 0) 0x7f03f80ac640 con 0x7f03f80b1f60 2024-03-29T10:54:58.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:58.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:58.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.654+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e4004eb0 con 0x7f03f80b1f60 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 client.?.objecter ms_handle_connect 0x7f03f8143d10 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 client.?.objecter ms_handle_connect 0x7f03f8138a70 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673836133 0 0) 0x7f03f8132920 con 0x7f03f8143d10 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:58.637 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f80ac640 con 0x7f03f8143d10 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420920630 0 0) 0x7f03f81385c0 con 0x7f03f8138a70 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:54:58.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f8132920 con 0x7f03f8138a70 2024-03-29T10:54:58.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968495584 0 0) 0x7f03f8132920 con 0x7f03f8138a70 2024-03-29T10:54:58.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03f81385c0 con 0x7f03f8138a70 2024-03-29T10:54:58.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f03ec001620 con 0x7f03f8138a70 2024-03-29T10:54:58.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:58.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:58.643 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469498073 0 0) 0x7f03f80ac640 con 0x7f03f8143d10 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03f8132920 con 0x7f03f8143d10 2024-03-29T10:54:58.644 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828548746 0 0) 0x7f03e4004eb0 con 0x7f03f80b1f60 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03f80ac640 con 0x7f03f80b1f60 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f03f0001b90 con 0x7f03f8143d10 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:58.645 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f03e8004980 con 0x7f03f80b1f60 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:58.646 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2089370409 0 0) 0x7f03f81385c0 con 0x7f03f8138a70 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 >> v1:172.21.15.79:6790/0 conn(0x7f03f80b1f60 legacy=0x7f03f80a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 >> v1:172.21.15.82:6789/0 conn(0x7f03f8143d10 legacy=0x7f03f80aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 monclient: found mon.a 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80ab470 con 0x7f03f8138a70 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.655+0000 7f03fe83e640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 10 monclient: _renew_subs 2024-03-29T10:54:58.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f81403d0 con 0x7f03f8138a70 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 10 monclient: _renew_subs 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f80b0ba0 con 0x7f03f8138a70 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03fe83e640 1 librados: init done 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03ec001b10 con 0x7f03f8138a70 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f03ec0035f0 con 0x7f03f8138a70 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.656+0000 7f03f4ff9640 10 monclient: dump: 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:54:58.648 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.657+0000 7f03f5ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.659+0000 7f03fe83e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.659+0000 7f03fe83e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.659+0000 7f03fe83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.659+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d8005080 con 0x7f03f8138a70 2024-03-29T10:54:58.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.660+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f03ec003750 con 0x7f03f8138a70 2024-03-29T10:54:58.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f03ec0834d0 con 0x7f03f8138a70 2024-03-29T10:54:58.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f4ff9640 10 client.4473.objecter ms_dispatch 0x7f03f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:58.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f4ff9640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:58.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f4ff9640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:58.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f4ff9640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T10:54:58.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f7fff640 10 monclient: get_auth_request con 0x7f03e406e180 auth_method 0 2024-03-29T10:54:58.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.663+0000 7f03f4ff9640 10 client.4473.objecter ms_handle_connect 0x7f03e406e180 2024-03-29T10:54:58.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.667+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03ec04edb0 con 0x7f03f8138a70 2024-03-29T10:54:58.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.668+0000 7f03f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:58.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:58.668+0000 7f03f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:58.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.002+0000 7f03fe83e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:58.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.002+0000 7f03fe83e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:58.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.002+0000 7f03fe83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:54:58.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.002+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d8005330 con 0x7f03f8138a70 2024-03-29T10:54:58.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.003+0000 7f03f4ff9640 1 -- 172.21.15.79:0/3484725369 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f03ec050ce0 con 0x7f03f8138a70 2024-03-29T10:54:58.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.003+0000 7f03f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:58.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.003+0000 7f03f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:58.987 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:54:58.987 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:58.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.006+0000 7f03fe83e640 10 librados: watch_flush enter 2024-03-29T10:54:58.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.006+0000 7f03fe83e640 10 librados: watch_flush exit 2024-03-29T10:54:58.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.006+0000 7f03fe83e640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T10:54:58.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.006+0000 7f03fe83e640 10 client.4473.objecter successfully canceled tick 2024-03-29T10:54:58.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.006+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 >> v1:172.21.15.79:6816/4290694627 conn(0x7f03e406e180 legacy=0x7f03e4070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:58.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.007+0000 7f03fe83e640 10 monclient: shutdown 2024-03-29T10:54:58.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.007+0000 7f03fe83e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:58.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.007+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 >> v1:172.21.15.79:6789/0 conn(0x7f03f8138a70 legacy=0x7f03f8138f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:58.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.009+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 shutdown_connections 2024-03-29T10:54:58.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.009+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 >> 172.21.15.79:0/3484725369 conn(0x7f03f8066a00 msgr2=0x7f03f80b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:58.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.009+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 shutdown_connections 2024-03-29T10:54:58.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.009+0000 7f03fe83e640 1 -- 172.21.15.79:0/3484725369 wait complete. 2024-03-29T10:54:58.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:54:59.009+0000 7f03fe83e640 1 librados: shutdown 2024-03-29T10:54:59.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:05.814 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:05.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.994+0000 7fb1e681e640 10 monclient: get_monmap_and_config 2024-03-29T10:55:05.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.994+0000 7fb1e681e640 10 monclient: build_initial_monmap 2024-03-29T10:55:05.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.994+0000 7fb1e681e640 10 monclient: monmap: 2024-03-29T10:55:05.975 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:05.975 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:05.995410+0000 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:05.995410+0000 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:05.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.995+0000 7fb1e681e640 1 Processor -- start 2024-03-29T10:55:05.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.996+0000 7fb1e681e640 1 -- start start 2024-03-29T10:55:05.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.996+0000 7fb1e681e640 10 monclient: init 2024-03-29T10:55:05.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.996+0000 7fb1e681e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:05.978 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.996+0000 7fb1e681e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.997+0000 7fb1e681e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e01457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.997+0000 7fb1e681e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e013f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.997+0000 7fb1e681e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e00a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.997+0000 7fb1e681e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e00aaef0 con 0x7fb1e00a9d70 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.997+0000 7fb1e681e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e0145bd0 con 0x7fb1e013f650 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:05.999+0000 7fb1dffff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb1e01457f0 0x7fb1e013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:53308/0 (socket says 172.21.15.79:53308) 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.000+0000 7fb1dffff640 1 -- 172.21.15.79:0/2766906740 learned_addr learned my addr 172.21.15.79:0/2766906740 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:05.981 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.000+0000 7fb1e681e640 1 -- 172.21.15.79:0/2766906740 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e0145eb0 con 0x7fb1e01457f0 2024-03-29T10:55:05.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.000+0000 7fb1e681e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:05.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.000+0000 7fb1e681e640 10 monclient(hunting): authenticate will time out at 1425.292646s 2024-03-29T10:55:05.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822266561 0 0) 0x7fb1e0145eb0 con 0x7fb1e01457f0 2024-03-29T10:55:05.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:05.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:05.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:05.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:55:05.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8003520 con 0x7fb1e01457f0 2024-03-29T10:55:05.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2200842807 0 0) 0x7fb1c8003520 con 0x7fb1e01457f0 2024-03-29T10:55:05.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.001+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 >> v1:172.21.15.82:6789/0 conn(0x7fb1e013f650 legacy=0x7fb1e00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:05.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 >> v1:172.21.15.79:6789/0 conn(0x7fb1e00a9d70 legacy=0x7fb1e014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:05.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 1 monclient: found mon.noname-b 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e0146340 con 0x7fb1e01457f0 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb1d0001690 con 0x7fb1e01457f0 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1deffd640 10 monclient: dump: 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:05.987 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1e681e640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.003+0000 7fb1e681e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1d0001860 con 0x7fb1e01457f0 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1deffd640 1 -- 172.21.15.79:0/2766906740 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb1d0001cb0 con 0x7fb1e01457f0 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:05.988 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1deffd640 10 monclient: dump: 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1e681e640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:05.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.004+0000 7fb1e681e640 4 set_mon_vals no callback set 2024-03-29T10:55:05.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.006+0000 7fb1e681e640 10 monclient: shutdown 2024-03-29T10:55:05.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.006+0000 7fb1e681e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:05.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.006+0000 7fb1e681e640 1 -- 172.21.15.79:0/2766906740 >> v1:172.21.15.79:6790/0 conn(0x7fb1e01457f0 legacy=0x7fb1e013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:05.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.009+0000 7fb1e681e640 1 -- 172.21.15.79:0/2766906740 shutdown_connections 2024-03-29T10:55:05.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.009+0000 7fb1e681e640 1 -- 172.21.15.79:0/2766906740 >> 172.21.15.79:0/2766906740 conn(0x7fb1e0066a00 msgr2=0x7fb1e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:05.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.009+0000 7fb1e681e640 1 -- 172.21.15.79:0/2766906740 shutdown_connections 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.009+0000 7fb1e681e640 1 -- 172.21.15.79:0/2766906740 wait complete. 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.010+0000 7fb1e681e640 10 monclient: build_initial_monmap 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.010+0000 7fb1e681e640 10 monclient: monmap: 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:05.992 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:05.993 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 1 librados: starting msgr at 2024-03-29T10:55:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 1 librados: starting objecter 2024-03-29T10:55:05.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 1 Processor -- start 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 1 -- start start 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 1 librados: setting wanted keys 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 1 librados: calling monclient init 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.011+0000 7fb1e681e640 10 monclient: init 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e00a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e013f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:05.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e01457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e006e5b0 con 0x7fb1e00a9d70 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e0142570 con 0x7fb1e01457f0 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e01419c0 con 0x7fb1e013f650 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1e681e640 10 monclient(hunting): authenticate will time out at 1425.304794s 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1df7fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb1e013f650 0x7fb1e01404b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:33058/0 (socket says 172.21.15.79:33058) 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1df7fe640 1 -- 172.21.15.79:0/2684565857 learned_addr learned my addr 172.21.15.79:0/2684565857 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1dcff9640 10 client.?.objecter ms_handle_connect 0x7fb1e013f650 2024-03-29T10:55:05.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.012+0000 7fb1dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:05.999 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.018+0000 7fb1dcff9640 10 client.?.objecter ms_handle_connect 0x7fb1e00a9d70 2024-03-29T10:55:05.999 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.018+0000 7fb1dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.019+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287465183 0 0) 0x7fb1e006e5b0 con 0x7fb1e00a9d70 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.019+0000 7fb1dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.019+0000 7fb1dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.019+0000 7fb1dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.019+0000 7fb1dcff9640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.019+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1cc004eb0 con 0x7fb1e00a9d70 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874143674 0 0) 0x7fb1e01419c0 con 0x7fb1e013f650 2024-03-29T10:55:06.003 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e006e5b0 con 0x7fb1e013f650 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1506572537 0 0) 0x7fb1cc004eb0 con 0x7fb1e00a9d70 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e01419c0 con 0x7fb1e00a9d70 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb1d0001b70 con 0x7fb1e00a9d70 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): dump: 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:06.004 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1957239937 0 0) 0x7fb1e006e5b0 con 0x7fb1e013f650 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1cc004eb0 con 0x7fb1e013f650 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb1d4004100 con 0x7fb1e013f650 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:06.005 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.020+0000 7fb1dcff9640 10 monclient(hunting): dump: 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2472875682 0 0) 0x7fb1cc004eb0 con 0x7fb1e013f650 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 >> v1:172.21.15.82:6789/0 conn(0x7fb1e01457f0 legacy=0x7fb1e01e5e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:06.006 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 >> v1:172.21.15.79:6789/0 conn(0x7fb1e00a9d70 legacy=0x7fb1e013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 10 monclient: _finish_hunting 0 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 1 monclient: found mon.c 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e0141b50 con 0x7fb1e013f650 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 10 monclient: _finish_auth 0 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 10 monclient: _check_auth_tickets 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 10 monclient: _renew_subs 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e01442c0 con 0x7fb1e013f650 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 10 monclient: _renew_subs 2024-03-29T10:55:06.007 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:06.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e0141ea0 con 0x7fb1e013f650 2024-03-29T10:55:06.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.021+0000 7fb1e681e640 1 librados: init done 2024-03-29T10:55:06.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.022+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1d4000f50 con 0x7fb1e013f650 2024-03-29T10:55:06.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.022+0000 7fb1dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:06.008 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.022+0000 7fb1de7fc640 4 set_mon_vals no callback set 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb1d4001320 con 0x7fb1e013f650 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1dcff9640 10 monclient: dump: 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:06.010 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1e681e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1e681e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1e681e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.023+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c0005080 con 0x7fb1e013f650 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.025+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fb1d4001500 con 0x7fb1e013f650 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.026+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fb1d4084aa0 con 0x7fb1e013f650 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.026+0000 7fb1dcff9640 10 client.4553.objecter ms_dispatch 0x7fb1e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.026+0000 7fb1dcff9640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.026+0000 7fb1dcff9640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.026+0000 7fb1dcff9640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T10:55:06.011 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.026+0000 7fb1dffff640 10 monclient: get_auth_request con 0x7fb1cc06ddb0 auth_method 0 2024-03-29T10:55:06.012 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.027+0000 7fb1dcff9640 10 client.4553.objecter ms_handle_connect 0x7fb1cc06ddb0 2024-03-29T10:55:06.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.033+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1d4050380 con 0x7fb1e013f650 2024-03-29T10:55:06.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.033+0000 7fb1dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:06.014 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.033+0000 7fb1dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:06.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.367+0000 7fb1e681e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:06.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.367+0000 7fb1e681e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:06.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.367+0000 7fb1e681e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:06.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.367+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1c0005330 con 0x7fb1e013f650 2024-03-29T10:55:06.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.370+0000 7fb1dcff9640 1 -- 172.21.15.79:0/2684565857 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fb1d40522b0 con 0x7fb1e013f650 2024-03-29T10:55:06.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.370+0000 7fb1dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:06.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.370+0000 7fb1dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:06.353 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:06.353 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:06.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.374+0000 7fb1e681e640 10 librados: watch_flush enter 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.374+0000 7fb1e681e640 10 librados: watch_flush exit 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.374+0000 7fb1e681e640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.374+0000 7fb1e681e640 10 client.4553.objecter successfully canceled tick 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 >> v1:172.21.15.79:6816/4290694627 conn(0x7fb1cc06ddb0 legacy=0x7fb1cc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 10 monclient: shutdown 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 >> v1:172.21.15.79:6790/0 conn(0x7fb1e013f650 legacy=0x7fb1e01404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 shutdown_connections 2024-03-29T10:55:06.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 >> 172.21.15.79:0/2684565857 conn(0x7fb1e0066a00 msgr2=0x7fb1e013a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:06.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 shutdown_connections 2024-03-29T10:55:06.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.375+0000 7fb1e681e640 1 -- 172.21.15.79:0/2684565857 wait complete. 2024-03-29T10:55:06.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:06.378+0000 7fb1e681e640 1 librados: shutdown 2024-03-29T10:55:06.379 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:13.186 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:13.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.350+0000 7f7414d3d640 10 monclient: get_monmap_and_config 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.350+0000 7f7414d3d640 10 monclient: build_initial_monmap 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.350+0000 7f7414d3d640 10 monclient: monmap: 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:13.351781+0000 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:13.351781+0000 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:13.333 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.351+0000 7f7414d3d640 1 Processor -- start 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.352+0000 7f7414d3d640 1 -- start start 2024-03-29T10:55:13.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.352+0000 7f7414d3d640 10 monclient: init 2024-03-29T10:55:13.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:13.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:13.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f74100a9930 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f7410149dc0 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f741014ff40 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100aae60 con 0x7f741014ff40 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100b4310 con 0x7f7410149dc0 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100afdd0 con 0x7f74100a9930 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f7414d3d640 10 monclient(hunting): authenticate will time out at 1432.645389s 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f740e59c640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f74100a9930 0x7f7410143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:58302/0 (socket says 172.21.15.79:58302) 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.353+0000 7f740e59c640 1 -- 172.21.15.79:0/1764756605 learned_addr learned my addr 172.21.15.79:0/1764756605 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:13.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355165384 0 0) 0x7f74100afdd0 con 0x7f74100a9930 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f8003520 con 0x7f74100a9930 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153009219 0 0) 0x7f74100aae60 con 0x7f741014ff40 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74100afdd0 con 0x7f741014ff40 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547782399 0 0) 0x7f73f8003520 con 0x7f74100a9930 2024-03-29T10:55:13.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 >> v1:172.21.15.82:6789/0 conn(0x7f7410149dc0 legacy=0x7f741014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 >> v1:172.21.15.79:6789/0 conn(0x7f741014ff40 legacy=0x7f7410152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 monclient: found mon.noname-b 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741014a950 con 0x7f74100a9930 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: _finish_auth 0 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f7414d3d640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f7414d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f74000015c0 con 0x7f74100a9930 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f740d59a640 10 monclient: dump: 2024-03-29T10:55:13.339 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.354+0000 7f7414d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7400001720 con 0x7f74100a9930 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f740d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:13.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f7414d3d640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f7414d3d640 4 set_mon_vals no callback set 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f740d59a640 1 -- 172.21.15.79:0/1764756605 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f7400001b30 con 0x7f74100a9930 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f740d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f740d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.355+0000 7f740d59a640 10 monclient: dump: 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:13.341 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.356+0000 7f7414d3d640 10 monclient: shutdown 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.356+0000 7f7414d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.356+0000 7f7414d3d640 1 -- 172.21.15.79:0/1764756605 >> v1:172.21.15.79:6790/0 conn(0x7f74100a9930 legacy=0x7f7410143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.356+0000 7f7414d3d640 1 -- 172.21.15.79:0/1764756605 shutdown_connections 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.356+0000 7f7414d3d640 1 -- 172.21.15.79:0/1764756605 >> 172.21.15.79:0/1764756605 conn(0x7f74100b35e0 msgr2=0x7f74100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.356+0000 7f7414d3d640 1 -- 172.21.15.79:0/1764756605 shutdown_connections 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.357+0000 7f7414d3d640 1 -- 172.21.15.79:0/1764756605 wait complete. 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 10 monclient: build_initial_monmap 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 10 monclient: monmap: 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:13.347 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 1 librados: starting msgr at 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 1 librados: starting objecter 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 1 Processor -- start 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 1 -- start start 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 1 librados: setting wanted keys 2024-03-29T10:55:13.348 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 1 librados: calling monclient init 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.358+0000 7f7414d3d640 10 monclient: init 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.359+0000 7f7414d3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.359+0000 7f7414d3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.359+0000 7f7414d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f74100a9930 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.359+0000 7f7414d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f7410149dc0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.359+0000 7f7414d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f741014ff40 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f7414d3d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74101dba80 con 0x7f7410149dc0 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f7414d3d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741006fb60 con 0x7f74100a9930 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f740e59c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f74100a9930 0x7f74101dbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:45052/0 (socket says 172.21.15.79:45052) 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f740e59c640 1 -- 172.21.15.79:0/623391160 learned_addr learned my addr 172.21.15.79:0/623391160 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 client.?.objecter ms_handle_connect 0x7f74100a9930 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 client.?.objecter ms_handle_connect 0x7f741014ff40 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:13.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741006fcf0 con 0x7f741014ff40 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f7414d3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f7414d3d640 10 monclient(hunting): authenticate will time out at 1432.652528s 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268007810 0 0) 0x7f741006fb60 con 0x7f74100a9930 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f0004eb0 con 0x7f74100a9930 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 client.?.objecter ms_handle_connect 0x7f7410149dc0 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.360+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 725135854 0 0) 0x7f73f0004eb0 con 0x7f74100a9930 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f741006fb60 con 0x7f74100a9930 2024-03-29T10:55:13.350 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f7400001b30 con 0x7f74100a9930 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): dump: 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:13.351 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997009268 0 0) 0x7f74101dba80 con 0x7f7410149dc0 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f0004eb0 con 0x7f7410149dc0 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890471545 0 0) 0x7f741006fcf0 con 0x7f741014ff40 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:13.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74101dba80 con 0x7f741014ff40 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2663704508 0 0) 0x7f741006fb60 con 0x7f74100a9930 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.361+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 >> v1:172.21.15.79:6790/0 conn(0x7f741014ff40 legacy=0x7f74101e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 >> v1:172.21.15.79:6789/0 conn(0x7f7410149dc0 legacy=0x7f741006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 1 monclient: found mon.b 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74101db560 con 0x7f74100a9930 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:13.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 5 monclient: authenticate success, global_id 4450 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 10 monclient: _renew_subs 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74101db750 con 0x7f74100a9930 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 10 monclient: _renew_subs 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74100700f0 con 0x7f74100a9930 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f7414d3d640 1 librados: init done 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7400004a90 con 0x7f74100a9930 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f7400004e20 con 0x7f74100a9930 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:13.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73feffd640 10 monclient: dump: 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.362+0000 7f73fffff640 4 set_mon_vals no callback set 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.363+0000 7f7414d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.364+0000 7f7414d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.364+0000 7f7414d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:13.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.364+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f4005080 con 0x7f74100a9930 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.364+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f7400004f80 con 0x7f74100a9930 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.365+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f7400084bc0 con 0x7f74100a9930 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.365+0000 7f73feffd640 10 client.4450.objecter ms_dispatch 0x7f7410000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.365+0000 7f73feffd640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.365+0000 7f73feffd640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.365+0000 7f73feffd640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.365+0000 7f740dd9b640 10 monclient: get_auth_request con 0x7f73f006dea0 auth_method 0 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.366+0000 7f73feffd640 10 client.4450.objecter ms_handle_connect 0x7f73f006dea0 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.373+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74000504a0 con 0x7f74100a9930 2024-03-29T10:55:13.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.373+0000 7f73feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:13.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.373+0000 7f73feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:13.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.684+0000 7f7414d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:13.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.684+0000 7f7414d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:13.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.684+0000 7f7414d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:13.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.684+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73f4005330 con 0x7f74100a9930 2024-03-29T10:55:13.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.685+0000 7f73feffd640 1 -- 172.21.15.79:0/623391160 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f74000523d0 con 0x7f74100a9930 2024-03-29T10:55:13.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.685+0000 7f73feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:13.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.685+0000 7f73feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:13.668 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:13.668 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:13.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 10 librados: watch_flush enter 2024-03-29T10:55:13.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 10 librados: watch_flush exit 2024-03-29T10:55:13.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T10:55:13.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 10 client.4450.objecter successfully canceled tick 2024-03-29T10:55:13.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 >> v1:172.21.15.79:6816/4290694627 conn(0x7f73f006dea0 legacy=0x7f73f0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 10 monclient: shutdown 2024-03-29T10:55:13.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:13.672 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.689+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 >> v1:172.21.15.82:6789/0 conn(0x7f74100a9930 legacy=0x7f74101dbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:13.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.692+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 shutdown_connections 2024-03-29T10:55:13.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.692+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 >> 172.21.15.79:0/623391160 conn(0x7f74100b35e0 msgr2=0x7f74100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:13.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.693+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 shutdown_connections 2024-03-29T10:55:13.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.693+0000 7f7414d3d640 1 -- 172.21.15.79:0/623391160 wait complete. 2024-03-29T10:55:13.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:13.693+0000 7f7414d3d640 1 librados: shutdown 2024-03-29T10:55:13.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:20.499 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.669+0000 7fe9495e8640 10 monclient: get_monmap_and_config 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.669+0000 7fe9495e8640 10 monclient: build_initial_monmap 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.669+0000 7fe9495e8640 10 monclient: monmap: 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:20.670640+0000 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:20.670640+0000 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:20.651 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:20.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.670+0000 7fe9495e8640 1 Processor -- start 2024-03-29T10:55:20.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.670+0000 7fe9495e8640 1 -- start start 2024-03-29T10:55:20.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.670+0000 7fe9495e8640 10 monclient: init 2024-03-29T10:55:20.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.671+0000 7fe9495e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.671+0000 7fe9495e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.671+0000 7fe9495e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9441457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.671+0000 7fe9495e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe94413f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9495e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9440a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9495e8640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9440aaef0 con 0x7fe94413f650 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9495e8640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944145bd0 con 0x7fe9441457f0 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9495e8640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944145eb0 con 0x7fe9440a9d70 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9495e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9495e8640 10 monclient(hunting): authenticate will time out at 1439.964306s 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9437fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe9440a9d70 0x7fe94414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:45282/0 (socket says 172.21.15.79:45282) 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe9437fe640 1 -- 172.21.15.79:0/3627907626 learned_addr learned my addr 172.21.15.79:0/3627907626 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481124072 0 0) 0x7fe9440aaef0 con 0x7fe94413f650 2024-03-29T10:55:20.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c003520 con 0x7fe94413f650 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944815642 0 0) 0x7fe944145eb0 con 0x7fe9440a9d70 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.672+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9440aaef0 con 0x7fe9440a9d70 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1548516183 0 0) 0x7fe9440aaef0 con 0x7fe9440a9d70 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 >> v1:172.21.15.82:6789/0 conn(0x7fe9441457f0 legacy=0x7fe94413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 >> v1:172.21.15.79:6789/0 conn(0x7fe94413f650 legacy=0x7fe9440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:20.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 monclient: found mon.noname-b 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe944146340 con 0x7fe9440a9d70 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe93c0019f0 con 0x7fe9440a9d70 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: dump: 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.657 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe9495e8640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe9495e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe93c001b50 con 0x7fe9440a9d70 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 1 -- 172.21.15.79:0/3627907626 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe93c001f60 con 0x7fe9440a9d70 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe9495e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:20.658 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe941ffb640 10 monclient: dump: 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:20.659 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.673+0000 7fe9495e8640 4 set_mon_vals no callback set 2024-03-29T10:55:20.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.674+0000 7fe9495e8640 10 monclient: shutdown 2024-03-29T10:55:20.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.674+0000 7fe9495e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:20.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.674+0000 7fe9495e8640 1 -- 172.21.15.79:0/3627907626 >> v1:172.21.15.79:6790/0 conn(0x7fe9440a9d70 legacy=0x7fe94414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:20.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.675+0000 7fe9495e8640 1 -- 172.21.15.79:0/3627907626 shutdown_connections 2024-03-29T10:55:20.664 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.675+0000 7fe9495e8640 1 -- 172.21.15.79:0/3627907626 >> 172.21.15.79:0/3627907626 conn(0x7fe944066a00 msgr2=0x7fe944066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.675+0000 7fe9495e8640 1 -- 172.21.15.79:0/3627907626 shutdown_connections 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.675+0000 7fe9495e8640 1 -- 172.21.15.79:0/3627907626 wait complete. 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.676+0000 7fe9495e8640 10 monclient: build_initial_monmap 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.676+0000 7fe9495e8640 10 monclient: monmap: 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:20.665 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 1 librados: starting msgr at 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 1 librados: starting objecter 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 1 Processor -- start 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 1 -- start start 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 1 librados: setting wanted keys 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 1 librados: calling monclient init 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.677+0000 7fe9495e8640 10 monclient: init 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.678+0000 7fe9495e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.678+0000 7fe9495e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.678+0000 7fe9495e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9440a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.678+0000 7fe9495e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe94413f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.678+0000 7fe9427fc640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe94413f650 0x7fe9441d77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:45602/0 (socket says 172.21.15.79:45602) 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.678+0000 7fe9427fc640 1 -- 172.21.15.79:0/271307631 learned_addr learned my addr 172.21.15.79:0/271307631 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:20.666 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe9495e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9441457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 client.?.objecter ms_handle_connect 0x7fe94413f650 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944142a90 con 0x7fe9441457f0 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 client.?.objecter ms_handle_connect 0x7fe9440a9d70 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944134e00 con 0x7fe9440a9d70 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 client.?.objecter ms_handle_connect 0x7fe9441457f0 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe944133e10 con 0x7fe94413f650 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929239711 0 0) 0x7fe944142a90 con 0x7fe9441457f0 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe9495e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe9495e8640 10 monclient(hunting): authenticate will time out at 1439.971927s 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:20.667 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.679+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe930004eb0 con 0x7fe9441457f0 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694813101 0 0) 0x7fe944133e10 con 0x7fe94413f650 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe944142a90 con 0x7fe94413f650 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452733575 0 0) 0x7fe944134e00 con 0x7fe9440a9d70 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe944133e10 con 0x7fe9440a9d70 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835389402 0 0) 0x7fe930004eb0 con 0x7fe9441457f0 2024-03-29T10:55:20.668 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe944134e00 con 0x7fe9441457f0 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe93c001e60 con 0x7fe9441457f0 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:20.669 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1844013548 0 0) 0x7fe944134e00 con 0x7fe9441457f0 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.680+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 >> v1:172.21.15.79:6790/0 conn(0x7fe94413f650 legacy=0x7fe9441d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 >> v1:172.21.15.82:6789/0 conn(0x7fe9440a9d70 legacy=0x7fe944133700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 1 monclient: found mon.a 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9441d7ef0 con 0x7fe9441457f0 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:20.670 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 5 monclient: authenticate success, global_id 4488 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 10 monclient: _renew_subs 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9441330e0 con 0x7fe9441457f0 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 10 monclient: _renew_subs 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe944132c50 con 0x7fe9441457f0 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe9495e8640 1 librados: init done 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe93c001fc0 con 0x7fe9441457f0 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe93c004fb0 con 0x7fe9441457f0 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe940ff9640 4 set_mon_vals no callback set 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:20.671 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.681+0000 7fe92b7fe640 10 monclient: dump: 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:20.672 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:20.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.683+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fe93c01af80 con 0x7fe9441457f0 2024-03-29T10:55:20.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.683+0000 7fe9495e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:20.673 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.683+0000 7fe9495e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.683+0000 7fe9495e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.683+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe924005080 con 0x7fe9441457f0 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.684+0000 7fe942ffd640 10 monclient: get_auth_request con 0x7fe93006da50 auth_method 0 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.685+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fe93c084dd0 con 0x7fe9441457f0 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.685+0000 7fe92b7fe640 10 client.4488.objecter ms_dispatch 0x7fe944000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.685+0000 7fe92b7fe640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.685+0000 7fe92b7fe640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.685+0000 7fe92b7fe640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.685+0000 7fe92b7fe640 10 client.4488.objecter ms_handle_connect 0x7fe93006da50 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.691+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe93c050710 con 0x7fe9441457f0 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.691+0000 7fe92b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:20.674 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:20.691+0000 7fe92b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:21.048 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.067+0000 7fe9495e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:21.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.067+0000 7fe9495e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:21.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.067+0000 7fe9495e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:21.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.067+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe924005330 con 0x7fe9441457f0 2024-03-29T10:55:21.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.069+0000 7fe92b7fe640 1 -- 172.21.15.79:0/271307631 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fe93c084480 con 0x7fe9441457f0 2024-03-29T10:55:21.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.069+0000 7fe92b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:21.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.069+0000 7fe92b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:21.050 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:21.050 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:21.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.072+0000 7fe9495e8640 10 librados: watch_flush enter 2024-03-29T10:55:21.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 10 librados: watch_flush exit 2024-03-29T10:55:21.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T10:55:21.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 10 client.4488.objecter successfully canceled tick 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 >> v1:172.21.15.79:6816/4290694627 conn(0x7fe93006da50 legacy=0x7fe93006ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 10 monclient: shutdown 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 >> v1:172.21.15.79:6789/0 conn(0x7fe9441457f0 legacy=0x7fe944135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 shutdown_connections 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 >> 172.21.15.79:0/271307631 conn(0x7fe944066a00 msgr2=0x7fe944130a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 shutdown_connections 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 -- 172.21.15.79:0/271307631 wait complete. 2024-03-29T10:55:21.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:21.073+0000 7fe9495e8640 1 librados: shutdown 2024-03-29T10:55:21.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:27.876 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:28.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.058+0000 7fad6a17a640 10 monclient: get_monmap_and_config 2024-03-29T10:55:28.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.058+0000 7fad6a17a640 10 monclient: build_initial_monmap 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.058+0000 7fad6a17a640 10 monclient: monmap: 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:28.059448+0000 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:28.059448+0000 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:28.040 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.060+0000 7fad6a17a640 1 Processor -- start 2024-03-29T10:55:28.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.060+0000 7fad6a17a640 1 -- start start 2024-03-29T10:55:28.041 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.060+0000 7fad6a17a640 10 monclient: init 2024-03-29T10:55:28.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:28.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient(hunting): picked mon.noname-a con 0x7fad641457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient(hunting): picked mon.noname-b con 0x7fad6413f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient(hunting): picked mon.noname-c con 0x7fad640a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad640aaef0 con 0x7fad641457f0 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad64145bd0 con 0x7fad640a9d70 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad64145eb0 con 0x7fad6413f650 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:28.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.061+0000 7fad6a17a640 10 monclient(hunting): authenticate will time out at 1447.353891s 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.062+0000 7fad62ffd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fad6413f650 0x7fad640b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:37654/0 (socket says 172.21.15.79:37654) 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.062+0000 7fad62ffd640 1 -- 172.21.15.79:0/3620627936 learned_addr learned my addr 172.21.15.79:0/3620627936 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617444496 0 0) 0x7fad64145eb0 con 0x7fad6413f650 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c003520 con 0x7fad6413f650 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 79452264 0 0) 0x7fad4c003520 con 0x7fad6413f650 2024-03-29T10:55:28.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.063+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 >> v1:172.21.15.82:6789/0 conn(0x7fad640a9d70 legacy=0x7fad6414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.065+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 >> v1:172.21.15.79:6789/0 conn(0x7fad641457f0 legacy=0x7fad6413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64146340 con 0x7fad6413f650 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad6a17a640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad6a17a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fad5c001a10 con 0x7fad6413f650 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:28.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad627fc640 10 monclient: dump: 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.066+0000 7fad6a17a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad5c001ca0 con 0x7fad6413f650 2024-03-29T10:55:28.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad6a17a640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad6a17a640 4 set_mon_vals no callback set 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad627fc640 1 -- 172.21.15.79:0/3620627936 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fad5c002030 con 0x7fad6413f650 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.067+0000 7fad627fc640 10 monclient: dump: 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.051 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.068+0000 7fad6a17a640 10 monclient: shutdown 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.068+0000 7fad6a17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.068+0000 7fad6a17a640 1 -- 172.21.15.79:0/3620627936 >> v1:172.21.15.79:6790/0 conn(0x7fad6413f650 legacy=0x7fad640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.068+0000 7fad6a17a640 1 -- 172.21.15.79:0/3620627936 shutdown_connections 2024-03-29T10:55:28.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.068+0000 7fad6a17a640 1 -- 172.21.15.79:0/3620627936 >> 172.21.15.79:0/3620627936 conn(0x7fad64066a00 msgr2=0x7fad64066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:28.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.072+0000 7fad6a17a640 1 -- 172.21.15.79:0/3620627936 shutdown_connections 2024-03-29T10:55:28.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.072+0000 7fad6a17a640 1 -- 172.21.15.79:0/3620627936 wait complete. 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.073+0000 7fad6a17a640 10 monclient: build_initial_monmap 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.073+0000 7fad6a17a640 10 monclient: monmap: 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.054 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:28.055 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:28.055 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:28.055 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.074+0000 7fad6a17a640 1 librados: starting msgr at 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.074+0000 7fad6a17a640 1 librados: starting objecter 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.074+0000 7fad6a17a640 1 Processor -- start 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.074+0000 7fad6a17a640 1 -- start start 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.074+0000 7fad6a17a640 1 librados: setting wanted keys 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.074+0000 7fad6a17a640 1 librados: calling monclient init 2024-03-29T10:55:28.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.075+0000 7fad6a17a640 10 monclient: init 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.075+0000 7fad6a17a640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 10 monclient(hunting): picked mon.noname-a con 0x7fad640a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 10 monclient(hunting): picked mon.noname-b con 0x7fad6413f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 10 monclient(hunting): picked mon.noname-c con 0x7fad641457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad641396a0 con 0x7fad640a9d70 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6413a390 con 0x7fad6413f650 2024-03-29T10:55:28.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6413a600 con 0x7fad641457f0 2024-03-29T10:55:28.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad63fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fad641457f0 0x7fad64138f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:40286/0 (socket says 172.21.15.79:40286) 2024-03-29T10:55:28.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad63fff640 1 -- 172.21.15.79:0/1852853578 learned_addr learned my addr 172.21.15.79:0/1852853578 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:28.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:28.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.076+0000 7fad6a17a640 10 monclient(hunting): authenticate will time out at 1447.368922s 2024-03-29T10:55:28.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.077+0000 7fad4bfff640 10 client.?.objecter ms_handle_connect 0x7fad641457f0 2024-03-29T10:55:28.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.077+0000 7fad4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920826522 0 0) 0x7fad6413a600 con 0x7fad641457f0 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c003520 con 0x7fad641457f0 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.080+0000 7fad4bfff640 10 client.?.objecter ms_handle_connect 0x7fad6413f650 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659514326 0 0) 0x7fad6413a390 con 0x7fad6413f650 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:28.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad6413a600 con 0x7fad6413f650 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639096705 0 0) 0x7fad4c003520 con 0x7fad641457f0 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad6413a390 con 0x7fad641457f0 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fad540017f0 con 0x7fad641457f0 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): dump: 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:28.067 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.081+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1815128922 0 0) 0x7fad6413a600 con 0x7fad6413f650 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad4c003520 con 0x7fad6413f650 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fad5c0018d0 con 0x7fad6413f650 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 10 monclient(hunting): dump: 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:28.068 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 266809459 0 0) 0x7fad4c003520 con 0x7fad6413f650 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 >> v1:172.21.15.79:6790/0 conn(0x7fad641457f0 legacy=0x7fad64138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 >> v1:172.21.15.79:6789/0 conn(0x7fad640a9d70 legacy=0x7fad6413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 1 monclient: found mon.b 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.082+0000 7fad4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad4bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:28.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64138a90 con 0x7fad6413f650 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad4bfff640 10 monclient: _finish_auth 0 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad4bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 5 monclient: authenticate success, global_id 4456 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 10 monclient: _renew_subs 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad641d7010 con 0x7fad6413f650 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 10 monclient: _renew_subs 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad641d42d0 con 0x7fad6413f650 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad6a17a640 1 librados: init done 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.083+0000 7fad4bfff640 10 client.4456.objecter ms_handle_connect 0x7fad640a9d70 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.085+0000 7fad4bfff640 10 client.4456.objecter resend_mon_ops 2024-03-29T10:55:28.070 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.085+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad5c001ab0 con 0x7fad6413f650 2024-03-29T10:55:28.071 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.085+0000 7fad4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:28.072 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.085+0000 7fad61ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fad5c005110 con 0x7fad6413f650 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad4bfff640 10 monclient: dump: 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:28.073 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad6a17a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad6a17a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad6a17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.086+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad44005080 con 0x7fad6413f650 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.087+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fad5c01b0e0 con 0x7fad6413f650 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.088+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fad5c084e50 con 0x7fad6413f650 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.088+0000 7fad637fe640 10 monclient: get_auth_request con 0x7fad4c06e0a0 auth_method 0 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.089+0000 7fad4bfff640 10 client.4456.objecter ms_dispatch 0x7fad64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.089+0000 7fad4bfff640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.089+0000 7fad4bfff640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.089+0000 7fad4bfff640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T10:55:28.074 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.089+0000 7fad4bfff640 10 client.4456.objecter ms_handle_connect 0x7fad4c06e0a0 2024-03-29T10:55:28.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.094+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad5c050730 con 0x7fad6413f650 2024-03-29T10:55:28.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.094+0000 7fad4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:28.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.094+0000 7fad4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.435+0000 7fad6a17a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.435+0000 7fad6a17a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.435+0000 7fad6a17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.435+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad44005330 con 0x7fad6413f650 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.436+0000 7fad4bfff640 1 -- 172.21.15.79:0/1852853578 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fad5c052660 con 0x7fad6413f650 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.436+0000 7fad4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:28.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.436+0000 7fad4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:28.418 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:28.418 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:28.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.439+0000 7fad6a17a640 10 librados: watch_flush enter 2024-03-29T10:55:28.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 10 librados: watch_flush exit 2024-03-29T10:55:28.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 10 client.4456.objecter successfully canceled tick 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 >> v1:172.21.15.79:6816/4290694627 conn(0x7fad4c06e0a0 legacy=0x7fad4c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 10 monclient: shutdown 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 >> v1:172.21.15.82:6789/0 conn(0x7fad6413f650 legacy=0x7fad641426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 shutdown_connections 2024-03-29T10:55:28.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.440+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 >> 172.21.15.79:0/1852853578 conn(0x7fad64066a00 msgr2=0x7fad640b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:28.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.441+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 shutdown_connections 2024-03-29T10:55:28.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.444+0000 7fad6a17a640 1 -- 172.21.15.79:0/1852853578 wait complete. 2024-03-29T10:55:28.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:28.444+0000 7fad6a17a640 1 librados: shutdown 2024-03-29T10:55:28.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:35.249 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.413+0000 7fb47d18c640 10 monclient: get_monmap_and_config 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.413+0000 7fb47d18c640 10 monclient: build_initial_monmap 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.413+0000 7fb47d18c640 10 monclient: monmap: 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:35.414047+0000 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:35.414047+0000 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:35.394 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:35.395 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:35.395 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:35.395 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:35.395 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:35.397 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.416+0000 7fb47d18c640 1 Processor -- start 2024-03-29T10:55:35.398 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.417+0000 7fb47d18c640 1 -- start start 2024-03-29T10:55:35.398 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.417+0000 7fb47d18c640 10 monclient: init 2024-03-29T10:55:35.399 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.418+0000 7fb47d18c640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:35.399 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.418+0000 7fb47d18c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:35.399 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.418+0000 7fb47d18c640 10 monclient(hunting): picked mon.noname-a con 0x7fb4780b48a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:35.399 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.418+0000 7fb47d18c640 10 monclient(hunting): picked mon.noname-b con 0x7fb4780b3f70 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:35.400 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.419+0000 7fb47d18c640 10 monclient(hunting): picked mon.noname-c con 0x7fb47813fb20 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:35.400 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.419+0000 7fb47d18c640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780b43b0 con 0x7fb4780b48a0 2024-03-29T10:55:35.400 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.419+0000 7fb47d18c640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780b0b90 con 0x7fb47813fb20 2024-03-29T10:55:35.400 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.419+0000 7fb47d18c640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4780b19a0 con 0x7fb4780b3f70 2024-03-29T10:55:35.400 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.419+0000 7fb47d18c640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:35.400 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.419+0000 7fb47d18c640 10 monclient(hunting): authenticate will time out at 1454.711739s 2024-03-29T10:55:35.401 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.420+0000 7fb46ffff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb4780b3f70 0x7fb47813f410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:42252/0 (socket says 172.21.15.79:42252) 2024-03-29T10:55:35.401 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.421+0000 7fb46ffff640 1 -- 172.21.15.79:0/990489962 learned_addr learned my addr 172.21.15.79:0/990489962 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310783651 0 0) 0x7fb4780b43b0 con 0x7fb4780b48a0 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient(hunting): _negotiate 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb464003520 con 0x7fb4780b48a0 2024-03-29T10:55:35.404 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 407831060 0 0) 0x7fb464003520 con 0x7fb4780b48a0 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 >> v1:172.21.15.79:6790/0 conn(0x7fb4780b3f70 legacy=0x7fb47813f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 >> v1:172.21.15.82:6789/0 conn(0x7fb47813fb20 legacy=0x7fb47814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: _finish_hunting 0 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 monclient: found mon.noname-a 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4780af800 con 0x7fb4780b48a0 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: _finish_auth 0 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: _check_auth_tickets 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb468001690 con 0x7fb4780b48a0 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:35.405 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47659c640 10 monclient: dump: 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47d18c640 5 monclient: authenticate success, global_id 4491 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.422+0000 7fb47d18c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb468001860 con 0x7fb4780b48a0 2024-03-29T10:55:35.406 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47659c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47659c640 1 -- 172.21.15.79:0/990489962 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb468001c70 con 0x7fb4780b48a0 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47659c640 10 monclient: dump: 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:35.407 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:35.408 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:35.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47d18c640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:35.408 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.423+0000 7fb47d18c640 4 set_mon_vals no callback set 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 10 monclient: shutdown 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 1 -- 172.21.15.79:0/990489962 >> v1:172.21.15.79:6789/0 conn(0x7fb4780b48a0 legacy=0x7fb4780b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 1 -- 172.21.15.79:0/990489962 shutdown_connections 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 1 -- 172.21.15.79:0/990489962 >> 172.21.15.79:0/990489962 conn(0x7fb47806c8a0 msgr2=0x7fb47806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 1 -- 172.21.15.79:0/990489962 shutdown_connections 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.424+0000 7fb47d18c640 1 -- 172.21.15.79:0/990489962 wait complete. 2024-03-29T10:55:35.409 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.425+0000 7fb47d18c640 10 monclient: build_initial_monmap 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.425+0000 7fb47d18c640 10 monclient: monmap: 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.426+0000 7fb47d18c640 1 librados: starting msgr at 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.426+0000 7fb47d18c640 1 librados: starting objecter 2024-03-29T10:55:35.410 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.426+0000 7fb47d18c640 1 Processor -- start 2024-03-29T10:55:35.411 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.429+0000 7fb47d18c640 1 -- start start 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.429+0000 7fb47d18c640 1 librados: setting wanted keys 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.429+0000 7fb47d18c640 1 librados: calling monclient init 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.429+0000 7fb47d18c640 10 monclient: init 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.429+0000 7fb47d18c640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.430+0000 7fb47d18c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.430+0000 7fb47d18c640 10 monclient(hunting): picked mon.noname-c con 0x7fb4780b3f70 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.430+0000 7fb476d9d640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fb4780b3f70 0x7fb478139ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43756/0 (socket says 172.21.15.79:43756) 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.430+0000 7fb476d9d640 1 -- 172.21.15.79:0/3390273169 learned_addr learned my addr 172.21.15.79:0/3390273169 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.430+0000 7fb46f7fe640 10 client.?.objecter ms_handle_connect 0x7fb4780b3f70 2024-03-29T10:55:35.412 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.430+0000 7fb46f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:35.414 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.433+0000 7fb47d18c640 10 monclient(hunting): picked mon.noname-a con 0x7fb4780b48a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:35.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb47d18c640 10 monclient(hunting): picked mon.noname-b con 0x7fb47813fb20 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:35.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb478142cd0 con 0x7fb4780b48a0 2024-03-29T10:55:35.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4781d6e40 con 0x7fb47813fb20 2024-03-29T10:55:35.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47813ff70 con 0x7fb4780b3f70 2024-03-29T10:55:35.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb47d18c640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:35.415 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb47d18c640 10 monclient(hunting): authenticate will time out at 1454.726616s 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357317572 0 0) 0x7fb47813ff70 con 0x7fb4780b3f70 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb46f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb46f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb46f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.434+0000 7fb46f7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb458004eb0 con 0x7fb4780b3f70 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 10 client.?.objecter ms_handle_connect 0x7fb4780b48a0 2024-03-29T10:55:35.416 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381826809 0 0) 0x7fb458004eb0 con 0x7fb4780b3f70 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb47813ff70 con 0x7fb4780b3f70 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb468001bb0 con 0x7fb4780b3f70 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:35.417 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.435+0000 7fb46f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.436+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2647339433 0 0) 0x7fb47813ff70 con 0x7fb4780b3f70 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.436+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 >> v1:172.21.15.82:6789/0 conn(0x7fb47813fb20 legacy=0x7fb478144f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:35.418 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 >> v1:172.21.15.79:6789/0 conn(0x7fb4780b48a0 legacy=0x7fb478144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:35.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:35.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 1 monclient: found mon.c 2024-03-29T10:55:35.419 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:35.420 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4781d7360 con 0x7fb4780b3f70 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb46f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.438+0000 7fb47d18c640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 10 monclient: _renew_subs 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47813a5b0 con 0x7fb4780b3f70 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 10 monclient: _renew_subs 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4781d6a20 con 0x7fb4780b3f70 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47d18c640 1 librados: init done 2024-03-29T10:55:35.421 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb468003960 con 0x7fb4780b3f70 2024-03-29T10:55:35.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb46f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:35.422 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.439+0000 7fb47559a640 4 set_mon_vals no callback set 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.440+0000 7fb47d18c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.440+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fb468001270 con 0x7fb4780b3f70 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.440+0000 7fb46f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.440+0000 7fb46f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.440+0000 7fb46f7fe640 10 monclient: dump: 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:35.423 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:35.424 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:35.424 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:35.424 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:35.424 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:35.424 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:35.424 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.444+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fb468003d70 con 0x7fb4780b3f70 2024-03-29T10:55:35.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.444+0000 7fb47d18c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:35.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.444+0000 7fb47d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:35.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.444+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c005080 con 0x7fb4780b3f70 2024-03-29T10:55:35.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.445+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fb468084bb0 con 0x7fb4780b3f70 2024-03-29T10:55:35.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.445+0000 7fb46f7fe640 10 client.4574.objecter ms_dispatch 0x7fb478000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:35.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.445+0000 7fb46f7fe640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:35.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.445+0000 7fb46f7fe640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:35.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.445+0000 7fb46f7fe640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T10:55:35.426 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.445+0000 7fb46ffff640 10 monclient: get_auth_request con 0x7fb45806dbd0 auth_method 0 2024-03-29T10:55:35.427 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.446+0000 7fb46f7fe640 10 client.4574.objecter ms_handle_connect 0x7fb45806dbd0 2024-03-29T10:55:35.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.454+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb468050490 con 0x7fb4780b3f70 2024-03-29T10:55:35.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.454+0000 7fb46f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:35.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.454+0000 7fb46f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:35.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.785+0000 7fb47d18c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:35.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.785+0000 7fb47d18c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:35.768 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.785+0000 7fb47d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:35.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.785+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb45c005330 con 0x7fb4780b3f70 2024-03-29T10:55:35.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.787+0000 7fb46f7fe640 1 -- 172.21.15.79:0/3390273169 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fb4680523c0 con 0x7fb4780b3f70 2024-03-29T10:55:35.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.787+0000 7fb46f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:35.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.787+0000 7fb46f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:35.769 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:35.769 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:35.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 10 librados: watch_flush enter 2024-03-29T10:55:35.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 10 librados: watch_flush exit 2024-03-29T10:55:35.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T10:55:35.773 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 10 client.4574.objecter successfully canceled tick 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 >> v1:172.21.15.79:6816/4290694627 conn(0x7fb45806dbd0 legacy=0x7fb458070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 10 monclient: shutdown 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 >> v1:172.21.15.79:6790/0 conn(0x7fb4780b3f70 legacy=0x7fb478139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 shutdown_connections 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 >> 172.21.15.79:0/3390273169 conn(0x7fb47806c8a0 msgr2=0x7fb47813f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.791+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 shutdown_connections 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.792+0000 7fb47d18c640 1 -- 172.21.15.79:0/3390273169 wait complete. 2024-03-29T10:55:35.774 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:35.792+0000 7fb47d18c640 1 librados: shutdown 2024-03-29T10:55:35.793 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:42.600 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.778+0000 7f2c67fff640 10 monclient: get_monmap_and_config 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.779+0000 7f2c67fff640 10 monclient: build_initial_monmap 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.779+0000 7f2c67fff640 10 monclient: monmap: 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:42.779939+0000 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:42.779939+0000 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:42.760 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:42.761 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:42.761 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:42.763 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.782+0000 7f2c67fff640 1 Processor -- start 2024-03-29T10:55:42.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.784+0000 7f2c67fff640 1 -- start start 2024-03-29T10:55:42.766 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.784+0000 7f2c67fff640 10 monclient: init 2024-03-29T10:55:42.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.785+0000 7f2c67fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:42.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.785+0000 7f2c67fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:42.767 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.785+0000 7f2c67fff640 10 monclient(hunting): picked mon.noname-b con 0x7f2c600389e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:42.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c667fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f2c60038dc0 0x7f2c600cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:53227/0 (socket says 172.21.15.79:53227) 2024-03-29T10:55:42.769 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c667fc640 1 -- 172.21.15.79:0/466415547 learned_addr learned my addr 172.21.15.79:0/466415547 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c67fff640 10 monclient(hunting): picked mon.noname-a con 0x7f2c60038dc0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c67fff640 10 monclient(hunting): picked mon.noname-c con 0x7f2c60030660 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c600c54c0 con 0x7f2c60038dc0 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c60036430 con 0x7f2c60030660 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.788+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c60039840 con 0x7f2c600389e0 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c67fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c67fff640 10 monclient(hunting): authenticate will time out at 1462.081215s 2024-03-29T10:55:42.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632950156 0 0) 0x7f2c600c54c0 con 0x7f2c60038dc0 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c65ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c65ffb640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.789+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c50003520 con 0x7f2c60038dc0 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.790+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283132564 0 0) 0x7f2c50003520 con 0x7f2c60038dc0 2024-03-29T10:55:42.771 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.790+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 >> v1:172.21.15.79:6790/0 conn(0x7f2c600389e0 legacy=0x7f2c60033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:42.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.792+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 >> v1:172.21.15.82:6789/0 conn(0x7f2c60030660 legacy=0x7f2c600d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 1 monclient: found mon.noname-a 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60035690 con 0x7f2c60038dc0 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2c5c001690 con 0x7f2c60038dc0 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c67fff640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c67fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:42.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c65ffb640 10 monclient: dump: 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:42.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.794+0000 7f2c67fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:42.778 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.797+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c5c001920 con 0x7f2c60038dc0 2024-03-29T10:55:42.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.797+0000 7f2c65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:42.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.798+0000 7f2c65ffb640 1 -- 172.21.15.79:0/466415547 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2c5c001cb0 con 0x7f2c60038dc0 2024-03-29T10:55:42.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.798+0000 7f2c67fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:42.779 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.798+0000 7f2c67fff640 4 set_mon_vals no callback set 2024-03-29T10:55:42.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.799+0000 7f2c67fff640 10 monclient: shutdown 2024-03-29T10:55:42.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.799+0000 7f2c67fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:42.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.799+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 >> v1:172.21.15.79:6789/0 conn(0x7f2c60038dc0 legacy=0x7f2c600cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:42.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.799+0000 7f2c65ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:55:42.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.799+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 shutdown_connections 2024-03-29T10:55:42.780 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.799+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 >> 172.21.15.79:0/466415547 conn(0x7f2c60005c00 msgr2=0x7f2c60006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:42.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.803+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 shutdown_connections 2024-03-29T10:55:42.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.805+0000 7f2c67fff640 1 -- 172.21.15.79:0/466415547 wait complete. 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.806+0000 7f2c67fff640 10 monclient: build_initial_monmap 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.806+0000 7f2c67fff640 10 monclient: monmap: 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:42.788 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 1 librados: starting msgr at 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 1 librados: starting objecter 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 1 Processor -- start 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 1 -- start start 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 1 librados: setting wanted keys 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 1 librados: calling monclient init 2024-03-29T10:55:42.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.807+0000 7f2c67fff640 10 monclient: init 2024-03-29T10:55:42.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.808+0000 7f2c67fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:42.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.808+0000 7f2c67fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:42.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.808+0000 7f2c67fff640 10 monclient(hunting): picked mon.noname-c con 0x7f2c60030660 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:42.792 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.811+0000 7f2c67fff640 10 monclient(hunting): picked mon.noname-a con 0x7f2c600389e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:42.792 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.811+0000 7f2c667fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f2c600389e0 0x7f2c600c59f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:54635/0 (socket says 172.21.15.79:54635) 2024-03-29T10:55:42.793 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c67fff640 10 monclient(hunting): picked mon.noname-b con 0x7f2c6002f590 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:42.793 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c667fc640 1 -- 172.21.15.79:0/2339110397 learned_addr learned my addr 172.21.15.79:0/2339110397 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:42.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c600c7090 con 0x7f2c600389e0 2024-03-29T10:55:42.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6015d660 con 0x7f2c6002f590 2024-03-29T10:55:42.797 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c601685f0 con 0x7f2c60030660 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c67fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.813+0000 7f2c67fff640 10 monclient(hunting): authenticate will time out at 1462.105822s 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.814+0000 7f2c4f7fe640 10 client.?.objecter ms_handle_connect 0x7f2c600389e0 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.814+0000 7f2c4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215903596 0 0) 0x7f2c600c7090 con 0x7f2c600389e0 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:55:42.798 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c54004eb0 con 0x7f2c600389e0 2024-03-29T10:55:42.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 10 client.?.objecter ms_handle_connect 0x7f2c6002f590 2024-03-29T10:55:42.799 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.816+0000 7f2c4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:42.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.819+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602505529 0 0) 0x7f2c6015d660 con 0x7f2c6002f590 2024-03-29T10:55:42.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.819+0000 7f2c4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:42.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.819+0000 7f2c4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:42.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.819+0000 7f2c4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:42.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.819+0000 7f2c4f7fe640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:55:42.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.819+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c600c7090 con 0x7f2c6002f590 2024-03-29T10:55:42.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.820+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 89953649 0 0) 0x7f2c54004eb0 con 0x7f2c600389e0 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.820+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c6015d660 con 0x7f2c600389e0 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.820+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2c5c001610 con 0x7f2c600389e0 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.820+0000 7f2c4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.820+0000 7f2c4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:42.804 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1072385769 0 0) 0x7f2c600c7090 con 0x7f2c6002f590 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c54004eb0 con 0x7f2c6002f590 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 350532216 0 0) 0x7f2c6015d660 con 0x7f2c600389e0 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 >> v1:172.21.15.79:6790/0 conn(0x7f2c60030660 legacy=0x7f2c6015cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 >> v1:172.21.15.82:6789/0 conn(0x7f2c6002f590 legacy=0x7f2c600c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 monclient: found mon.a 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60168780 con 0x7f2c600389e0 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:42.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 10 monclient: _renew_subs 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c601689d0 con 0x7f2c600389e0 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 10 monclient: _renew_subs 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c60168d40 con 0x7f2c600389e0 2024-03-29T10:55:42.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.821+0000 7f2c67fff640 1 librados: init done 2024-03-29T10:55:42.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.823+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c5c001770 con 0x7f2c600389e0 2024-03-29T10:55:42.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.823+0000 7f2c4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.824+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2c5c004d90 con 0x7f2c600389e0 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.824+0000 7f2c4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.824+0000 7f2c4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.824+0000 7f2c4f7fe640 10 monclient: dump: 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:42.808 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.824+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f2c5c004fb0 con 0x7f2c600389e0 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.824+0000 7f2c657fa640 4 set_mon_vals no callback set 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.825+0000 7f2c67fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.825+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f2c5c084c10 con 0x7f2c600389e0 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.825+0000 7f2c4f7fe640 10 client.4500.objecter ms_dispatch 0x7f2c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.825+0000 7f2c4f7fe640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.825+0000 7f2c4f7fe640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:42.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.825+0000 7f2c4f7fe640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:55:42.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.827+0000 7f2c67fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:42.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.827+0000 7f2c67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:42.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.827+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c48005080 con 0x7f2c600389e0 2024-03-29T10:55:42.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.828+0000 7f2c66ffd640 10 monclient: get_auth_request con 0x7f2c5406dd30 auth_method 0 2024-03-29T10:55:42.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.828+0000 7f2c4f7fe640 10 client.4500.objecter ms_handle_connect 0x7f2c5406dd30 2024-03-29T10:55:42.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.837+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2c5c0505b0 con 0x7f2c600389e0 2024-03-29T10:55:42.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.837+0000 7f2c4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:42.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:42.837+0000 7f2c4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:43.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.167+0000 7f2c67fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:43.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.167+0000 7f2c67fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:43.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.167+0000 7f2c67fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:43.148 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.167+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c48005330 con 0x7f2c600389e0 2024-03-29T10:55:43.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.168+0000 7f2c4f7fe640 1 -- 172.21.15.79:0/2339110397 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f2c5c0524e0 con 0x7f2c600389e0 2024-03-29T10:55:43.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.168+0000 7f2c4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:43.149 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.168+0000 7f2c4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:43.149 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:43.149 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.171+0000 7f2c67fff640 10 librados: watch_flush enter 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.171+0000 7f2c67fff640 10 librados: watch_flush exit 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.171+0000 7f2c67fff640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.171+0000 7f2c67fff640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.172+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 >> v1:172.21.15.79:6816/4290694627 conn(0x7f2c5406dd30 legacy=0x7f2c540701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.172+0000 7f2c67fff640 10 monclient: shutdown 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.172+0000 7f2c67fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:43.154 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.172+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 >> v1:172.21.15.79:6789/0 conn(0x7f2c600389e0 legacy=0x7f2c600c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:43.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.174+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 shutdown_connections 2024-03-29T10:55:43.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.174+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 >> 172.21.15.79:0/2339110397 conn(0x7f2c60005c00 msgr2=0x7f2c60034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:43.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.174+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 shutdown_connections 2024-03-29T10:55:43.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.174+0000 7f2c67fff640 1 -- 172.21.15.79:0/2339110397 wait complete. 2024-03-29T10:55:43.155 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:43.174+0000 7f2c67fff640 1 librados: shutdown 2024-03-29T10:55:43.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:49.982 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.151+0000 7fa61aab3640 10 monclient: get_monmap_and_config 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.151+0000 7fa61aab3640 10 monclient: build_initial_monmap 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.151+0000 7fa61aab3640 10 monclient: monmap: 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:50.152037+0000 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:50.152037+0000 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:50.133 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.134 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:50.134 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:50.134 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:50.134 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.134 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.152+0000 7fa61aab3640 1 Processor -- start 2024-03-29T10:55:50.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.152+0000 7fa61aab3640 1 -- start start 2024-03-29T10:55:50.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.152+0000 7fa61aab3640 10 monclient: init 2024-03-29T10:55:50.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.153+0000 7fa61aab3640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:50.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.153+0000 7fa61aab3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:50.135 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.153+0000 7fa61aab3640 10 monclient(hunting): picked mon.noname-b con 0x7fa60c034850 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:50.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.154+0000 7fa61aab3640 10 monclient(hunting): picked mon.noname-c con 0x7fa60c02a640 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:50.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa61aab3640 10 monclient(hunting): picked mon.noname-a con 0x7fa60c0327d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:50.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa61aab3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c02c820 con 0x7fa60c0327d0 2024-03-29T10:55:50.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa61aab3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c02a120 con 0x7fa60c02a640 2024-03-29T10:55:50.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa61aab3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c0c75b0 con 0x7fa60c034850 2024-03-29T10:55:50.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa61aab3640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:50.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa6192b0640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa60c02a640 0x7fa60c0320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:58931/0 (socket says 172.21.15.79:58931) 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.156+0000 7fa6192b0640 1 -- 172.21.15.79:0/1071922033 learned_addr learned my addr 172.21.15.79:0/1071922033 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa61aab3640 10 monclient(hunting): authenticate will time out at 1469.449116s 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110777178 0 0) 0x7fa60c02c820 con 0x7fa60c0327d0 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): _negotiate 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa600003520 con 0x7fa60c0327d0 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440966232 0 0) 0x7fa60c0c75b0 con 0x7fa60c034850 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): _negotiate 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:50.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.157+0000 7fa618aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa60c02c820 con 0x7fa60c034850 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498339946 0 0) 0x7fa600003520 con 0x7fa60c0327d0 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 >> v1:172.21.15.79:6790/0 conn(0x7fa60c034850 legacy=0x7fa60c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 >> v1:172.21.15.82:6789/0 conn(0x7fa60c02a640 legacy=0x7fa60c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 10 monclient: _finish_hunting 0 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 monclient: found mon.noname-a 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c033110 con 0x7fa60c0327d0 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 10 monclient: _finish_auth 0 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 10 monclient: _check_auth_tickets 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:50.143 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa604001690 con 0x7fa60c0327d0 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.158+0000 7fa618aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 10 monclient: dump: 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:50.144 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa604001920 con 0x7fa60c0327d0 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 1 -- 172.21.15.79:0/1071922033 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa604001cf0 con 0x7fa60c0327d0 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa618aaf640 10 monclient: dump: 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.145 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa61aab3640 5 monclient: authenticate success, global_id 4503 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa61aab3640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.159+0000 7fa61aab3640 4 set_mon_vals no callback set 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.160+0000 7fa61aab3640 10 monclient: shutdown 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.160+0000 7fa61aab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:50.146 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.160+0000 7fa61aab3640 1 -- 172.21.15.79:0/1071922033 >> v1:172.21.15.79:6789/0 conn(0x7fa60c0327d0 legacy=0x7fa60c0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.161+0000 7fa61aab3640 1 -- 172.21.15.79:0/1071922033 shutdown_connections 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.161+0000 7fa61aab3640 1 -- 172.21.15.79:0/1071922033 >> 172.21.15.79:0/1071922033 conn(0x7fa60c0058f0 msgr2=0x7fa60c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.161+0000 7fa61aab3640 1 -- 172.21.15.79:0/1071922033 shutdown_connections 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.161+0000 7fa61aab3640 1 -- 172.21.15.79:0/1071922033 wait complete. 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.162+0000 7fa61aab3640 10 monclient: build_initial_monmap 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.163+0000 7fa61aab3640 10 monclient: monmap: 2024-03-29T10:55:50.150 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.163+0000 7fa61aab3640 1 librados: starting msgr at 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.163+0000 7fa61aab3640 1 librados: starting objecter 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.163+0000 7fa61aab3640 1 Processor -- start 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 1 -- start start 2024-03-29T10:55:50.151 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 1 librados: setting wanted keys 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 1 librados: calling monclient init 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 10 monclient: init 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.164+0000 7fa61aab3640 10 monclient(hunting): picked mon.noname-c con 0x7fa60c02a640 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 10 monclient(hunting): picked mon.noname-a con 0x7fa60c0327d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 10 monclient(hunting): picked mon.noname-b con 0x7fa60c034850 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c02ce20 con 0x7fa60c0327d0 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c158ef0 con 0x7fa60c034850 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c158af0 con 0x7fa60c02a640 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa61aab3640 10 monclient(hunting): authenticate will time out at 1469.457395s 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa619ab1640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fa60c02a640 0x7fa60c1583e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:34974/0 (socket says 172.21.15.79:34974) 2024-03-29T10:55:50.152 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa619ab1640 1 -- 172.21.15.79:0/1964891102 learned_addr learned my addr 172.21.15.79:0/1964891102 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa60a7fc640 10 client.?.objecter ms_handle_connect 0x7fa60c02a640 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.165+0000 7fa60a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.166+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293572213 0 0) 0x7fa60c158af0 con 0x7fa60c02a640 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.166+0000 7fa60a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.166+0000 7fa60a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.166+0000 7fa60a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.166+0000 7fa60a7fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:55:50.153 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.166+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f8004eb0 con 0x7fa60c02a640 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917173988 0 0) 0x7fa5f8004eb0 con 0x7fa60c02a640 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa60c158af0 con 0x7fa60c02a640 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa61404fdd0 con 0x7fa60c02a640 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 10 monclient(hunting): dump: 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.160 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.169+0000 7fa60a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 client.?.objecter ms_handle_connect 0x7fa60c034850 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 client.?.objecter ms_handle_connect 0x7fa60c0327d0 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304839843 0 0) 0x7fa60c02ce20 con 0x7fa60c0327d0 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:55:50.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.170+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f80055e0 con 0x7fa60c0327d0 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248039911 0 0) 0x7fa60c158ef0 con 0x7fa60c034850 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa60c02ce20 con 0x7fa60c034850 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230655710 0 0) 0x7fa5f80055e0 con 0x7fa60c0327d0 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa60c158ef0 con 0x7fa60c0327d0 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa610000ed0 con 0x7fa60c0327d0 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): dump: 2024-03-29T10:55:50.162 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3947788944 0 0) 0x7fa60c02ce20 con 0x7fa60c034850 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5f80055e0 con 0x7fa60c034850 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa604001550 con 0x7fa60c034850 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:50.163 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.171+0000 7fa60a7fc640 10 monclient(hunting): dump: 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 248695937 0 0) 0x7fa60c158ef0 con 0x7fa60c0327d0 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 >> v1:172.21.15.79:6790/0 conn(0x7fa60c02a640 legacy=0x7fa60c1583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 >> v1:172.21.15.82:6789/0 conn(0x7fa60c034850 legacy=0x7fa60c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:50.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 1 monclient: found mon.a 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c158c80 con 0x7fa60c0327d0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa60a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 5 monclient: authenticate success, global_id 4506 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 10 monclient: _renew_subs 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c0bb520 con 0x7fa60c0327d0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 10 monclient: _renew_subs 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c0bf980 con 0x7fa60c0327d0 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.172+0000 7fa61aab3640 1 librados: init done 2024-03-29T10:55:50.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.173+0000 7fa61aab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.173+0000 7fa61aab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.173+0000 7fa61aab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.173+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c004cd0 con 0x7fa60c0327d0 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa610001300 con 0x7fa60c0327d0 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa610001690 con 0x7fa60c0327d0 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 10 monclient: dump: 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:50.166 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60bfff640 4 set_mon_vals no callback set 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.175+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fa6100018b0 con 0x7fa60c0327d0 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.176+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fa610084940 con 0x7fa60c0327d0 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.176+0000 7fa60a7fc640 10 client.4506.objecter ms_dispatch 0x7fa614000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.176+0000 7fa60a7fc640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.176+0000 7fa60a7fc640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.176+0000 7fa60a7fc640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.176+0000 7fa619ab1640 10 monclient: get_auth_request con 0x7fa5f806e350 auth_method 0 2024-03-29T10:55:50.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.177+0000 7fa60a7fc640 10 client.4506.objecter ms_handle_connect 0x7fa5f806e350 2024-03-29T10:55:50.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.185+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6100503a0 con 0x7fa60c0327d0 2024-03-29T10:55:50.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.185+0000 7fa60a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:50.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.185+0000 7fa60a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:50.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.510+0000 7fa61aab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:50.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.510+0000 7fa61aab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:50.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.510+0000 7fa61aab3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:55:50.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.510+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa60c159080 con 0x7fa60c0327d0 2024-03-29T10:55:50.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.511+0000 7fa60a7fc640 1 -- 172.21.15.79:0/1964891102 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fa6100522d0 con 0x7fa60c0327d0 2024-03-29T10:55:50.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.511+0000 7fa60a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:50.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.511+0000 7fa60a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:50.493 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:50.493 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 10 librados: watch_flush enter 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 10 librados: watch_flush exit 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 10 client.4506.objecter successfully canceled tick 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 >> v1:172.21.15.79:6816/4290694627 conn(0x7fa5f806e350 legacy=0x7fa5f8070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 10 monclient: shutdown 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:50.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.515+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 >> v1:172.21.15.79:6789/0 conn(0x7fa60c0327d0 legacy=0x7fa60c0bae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:50.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.516+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 shutdown_connections 2024-03-29T10:55:50.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.516+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 >> 172.21.15.79:0/1964891102 conn(0x7fa60c0058f0 msgr2=0x7fa60c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:50.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.517+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 shutdown_connections 2024-03-29T10:55:50.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.517+0000 7fa61aab3640 1 -- 172.21.15.79:0/1964891102 wait complete. 2024-03-29T10:55:50.499 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:50.517+0000 7fa61aab3640 1 librados: shutdown 2024-03-29T10:55:50.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:57.323 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:57.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.490+0000 7f4969ba3640 10 monclient: get_monmap_and_config 2024-03-29T10:55:57.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.490+0000 7f4969ba3640 10 monclient: build_initial_monmap 2024-03-29T10:55:57.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.490+0000 7f4969ba3640 10 monclient: monmap: 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:55:57.492405+0000 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:55:57.492405+0000 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:55:57.473 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.491+0000 7f4969ba3640 1 Processor -- start 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.491+0000 7f4969ba3640 1 -- start start 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.491+0000 7f4969ba3640 10 monclient: init 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f49640b0860 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f4964072450 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f496413ee40 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:57.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640ad840 con 0x7f49640b0860 2024-03-29T10:55:57.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.492+0000 7f4969ba3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496413c830 con 0x7f4964072450 2024-03-29T10:55:57.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.493+0000 7f4969ba3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496413c9c0 con 0x7f496413ee40 2024-03-29T10:55:57.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.493+0000 7f4969ba3640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:57.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.493+0000 7f4969ba3640 10 monclient(hunting): authenticate will time out at 1476.786251s 2024-03-29T10:55:57.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.494+0000 7f4963fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f496413ee40 0x7f496413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:42886/0 (socket says 172.21.15.79:42886) 2024-03-29T10:55:57.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.494+0000 7f4963fff640 1 -- 172.21.15.79:0/4090570283 learned_addr learned my addr 172.21.15.79:0/4090570283 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713988477 0 0) 0x7f496413c9c0 con 0x7f496413ee40 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4950005730 con 0x7f496413ee40 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3460044710 0 0) 0x7f4950005730 con 0x7f496413ee40 2024-03-29T10:55:57.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.496+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 >> v1:172.21.15.82:6789/0 conn(0x7f4964072450 legacy=0x7f496413c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 >> v1:172.21.15.79:6789/0 conn(0x7f49640b0860 legacy=0x7f49640acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496413f970 con 0x7f496413ee40 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:57.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f495c001670 con 0x7f496413ee40 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f49627fc640 10 monclient: dump: 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:57.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f4969ba3640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.497+0000 7f4969ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f495c001900 con 0x7f496413ee40 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f49627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f49627fc640 1 -- 172.21.15.79:0/4090570283 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f495c001cd0 con 0x7f496413ee40 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f49627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f49627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:57.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f49627fc640 10 monclient: dump: 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f4969ba3640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.498+0000 7f4969ba3640 4 set_mon_vals no callback set 2024-03-29T10:55:57.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.499+0000 7f4969ba3640 10 monclient: shutdown 2024-03-29T10:55:57.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.499+0000 7f4969ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:57.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.499+0000 7f4969ba3640 1 -- 172.21.15.79:0/4090570283 >> v1:172.21.15.79:6790/0 conn(0x7f496413ee40 legacy=0x7f496413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.500+0000 7f4969ba3640 1 -- 172.21.15.79:0/4090570283 shutdown_connections 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.500+0000 7f4969ba3640 1 -- 172.21.15.79:0/4090570283 >> 172.21.15.79:0/4090570283 conn(0x7f496406b660 msgr2=0x7f496406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.500+0000 7f4969ba3640 1 -- 172.21.15.79:0/4090570283 shutdown_connections 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.500+0000 7f4969ba3640 1 -- 172.21.15.79:0/4090570283 wait complete. 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.501+0000 7f4969ba3640 10 monclient: build_initial_monmap 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.502+0000 7f4969ba3640 10 monclient: monmap: 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:55:57.484 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.502+0000 7f4969ba3640 1 librados: starting msgr at 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.502+0000 7f4969ba3640 1 librados: starting objecter 2024-03-29T10:55:57.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.502+0000 7f4969ba3640 1 Processor -- start 2024-03-29T10:55:57.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.503+0000 7f4969ba3640 1 -- start start 2024-03-29T10:55:57.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.503+0000 7f4969ba3640 1 librados: setting wanted keys 2024-03-29T10:55:57.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.503+0000 7f4969ba3640 1 librados: calling monclient init 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.503+0000 7f4969ba3640 10 monclient: init 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f4964072450 addr v1:172.21.15.82:6789/0 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f49640b0860 addr v1:172.21.15.79:6789/0 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f496413ee40 addr v1:172.21.15.79:6790/0 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4964142c30 con 0x7f49640b0860 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4964142f10 con 0x7f4964072450 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49640b1350 con 0x7f496413ee40 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4969ba3640 10 monclient(hunting): authenticate will time out at 1476.797783s 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4963fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f496413ee40 0x7f49641ddb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43896/0 (socket says 172.21.15.79:43896) 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.504+0000 7f4963fff640 1 -- 172.21.15.79:0/3393838895 learned_addr learned my addr 172.21.15.79:0/3393838895 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 client.?.objecter ms_handle_connect 0x7f496413ee40 2024-03-29T10:55:57.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192392970 0 0) 0x7f49640b1350 con 0x7f496413ee40 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4944004eb0 con 0x7f496413ee40 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396607625 0 0) 0x7f4944004eb0 con 0x7f496413ee40 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49640b1350 con 0x7f496413ee40 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f495c001cd0 con 0x7f496413ee40 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): dump: 2024-03-29T10:55:57.490 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:57.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.505+0000 7f495bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:57.494 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 488911614 0 0) 0x7f49640b1350 con 0x7f496413ee40 2024-03-29T10:55:57.494 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 >> v1:172.21.15.82:6789/0 conn(0x7f4964072450 legacy=0x7f4964141070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:57.494 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 >> v1:172.21.15.79:6789/0 conn(0x7f49640b0860 legacy=0x7f4964136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:57.494 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:57.494 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 1 monclient: found mon.c 2024-03-29T10:55:57.494 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964141ba0 con 0x7f496413ee40 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 10 monclient: _finish_auth 0 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.509+0000 7f495bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 5 monclient: authenticate success, global_id 4586 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 10 monclient: _renew_subs 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49641372c0 con 0x7f496413ee40 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 10 monclient: _renew_subs 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49641420c0 con 0x7f496413ee40 2024-03-29T10:55:57.495 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f4969ba3640 1 librados: init done 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f495c004a90 con 0x7f496413ee40 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f495bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f495c004c80 con 0x7f496413ee40 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f495bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f495bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.510+0000 7f495bfff640 10 monclient: dump: 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:55:57.496 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:55:57.497 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:55:57.497 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:55:57.497 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:55:57.497 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.512+0000 7f49617fa640 4 set_mon_vals no callback set 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.513+0000 7f4969ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.513+0000 7f4969ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f4969ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f495c004e60 con 0x7f496413ee40 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4948005080 con 0x7f496413ee40 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f495c084bf0 con 0x7f496413ee40 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f495bfff640 10 client.4586.objecter ms_dispatch 0x7f4964000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f495bfff640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:57.498 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f495bfff640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:57.499 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.514+0000 7f495bfff640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T10:55:57.499 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.515+0000 7f49637fe640 10 monclient: get_auth_request con 0x7f494406db10 auth_method 0 2024-03-29T10:55:57.499 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.516+0000 7f495bfff640 10 client.4586.objecter ms_handle_connect 0x7f494406db10 2024-03-29T10:55:57.504 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.522+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f495c0504d0 con 0x7f496413ee40 2024-03-29T10:55:57.504 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.522+0000 7f495bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:57.504 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.522+0000 7f495bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:57.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.855+0000 7f4969ba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:57.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.855+0000 7f4969ba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:57.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.855+0000 7f4969ba3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:55:57.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.855+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4948005330 con 0x7f496413ee40 2024-03-29T10:55:57.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.858+0000 7f495bfff640 1 -- 172.21.15.79:0/3393838895 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f495c052400 con 0x7f496413ee40 2024-03-29T10:55:57.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.858+0000 7f495bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:57.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.858+0000 7f495bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:57.841 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:55:57.841 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:57.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 10 librados: watch_flush enter 2024-03-29T10:55:57.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 10 librados: watch_flush exit 2024-03-29T10:55:57.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T10:55:57.844 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 10 client.4586.objecter successfully canceled tick 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 >> v1:172.21.15.79:6816/4290694627 conn(0x7f494406db10 legacy=0x7f494406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 10 monclient: shutdown 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.862+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 >> v1:172.21.15.79:6790/0 conn(0x7f496413ee40 legacy=0x7f49641ddb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.863+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 shutdown_connections 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.863+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 >> 172.21.15.79:0/3393838895 conn(0x7f496406b660 msgr2=0x7f4964069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.863+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 shutdown_connections 2024-03-29T10:55:57.845 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.863+0000 7f4969ba3640 1 -- 172.21.15.79:0/3393838895 wait complete. 2024-03-29T10:55:57.848 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:55:57.866+0000 7f4969ba3640 1 librados: shutdown 2024-03-29T10:55:57.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:04.671 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.843+0000 7f8935c25640 10 monclient: get_monmap_and_config 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.843+0000 7f8935c25640 10 monclient: build_initial_monmap 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.843+0000 7f8935c25640 10 monclient: monmap: 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:04.844148+0000 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:04.844148+0000 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:04.824 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:04.825 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:04.825 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.844+0000 7f8935c25640 1 Processor -- start 2024-03-29T10:56:04.825 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.844+0000 7f8935c25640 1 -- start start 2024-03-29T10:56:04.825 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.844+0000 7f8935c25640 10 monclient: init 2024-03-29T10:56:04.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.845+0000 7f8935c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:04.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.845+0000 7f8935c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:04.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.845+0000 7f8935c25640 10 monclient(hunting): picked mon.noname-a con 0x7f8930074650 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:04.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.845+0000 7f8935c25640 10 monclient(hunting): picked mon.noname-c con 0x7f8930149ff0 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:04.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.845+0000 7f8935c25640 10 monclient(hunting): picked mon.noname-b con 0x7f8930072620 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:04.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.845+0000 7f8935c25640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893006ecf0 con 0x7f8930074650 2024-03-29T10:56:04.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.846+0000 7f8935c25640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8930070440 con 0x7f8930149ff0 2024-03-29T10:56:04.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.846+0000 7f8935c25640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8930070720 con 0x7f8930072620 2024-03-29T10:56:04.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.846+0000 7f8935c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:04.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.846+0000 7f8935c25640 10 monclient(hunting): authenticate will time out at 1484.138293s 2024-03-29T10:56:04.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892ffff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f8930072620 0x7f8930151c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35240/0 (socket says 172.21.15.79:35240) 2024-03-29T10:56:04.828 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892ffff640 1 -- 172.21.15.79:0/2928720832 learned_addr learned my addr 172.21.15.79:0/2928720832 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246555744 0 0) 0x7f8930070720 con 0x7f8930072620 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892effd640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.847+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c003520 con 0x7f8930072620 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.848+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200744938 0 0) 0x7f891c003520 con 0x7f8930072620 2024-03-29T10:56:04.829 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.848+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 >> v1:172.21.15.82:6789/0 conn(0x7f8930149ff0 legacy=0x7f8930071f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.850+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 >> v1:172.21.15.79:6789/0 conn(0x7f8930074650 legacy=0x7f89301498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 1 monclient: found mon.noname-b 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893006e730 con 0x7f8930072620 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: _finish_auth 0 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8928001a10 con 0x7f8930072620 2024-03-29T10:56:04.835 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f892effd640 10 monclient: dump: 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:04.836 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f8935c25640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.851+0000 7f8935c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8928001ca0 con 0x7f8930072620 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f892effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f892effd640 1 -- 172.21.15.79:0/2928720832 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8928002030 con 0x7f8930072620 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f892effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f892effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f892effd640 10 monclient: dump: 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.837 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f8935c25640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.852+0000 7f8935c25640 4 set_mon_vals no callback set 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.853+0000 7f8935c25640 10 monclient: shutdown 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.853+0000 7f8935c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.853+0000 7f8935c25640 1 -- 172.21.15.79:0/2928720832 >> v1:172.21.15.79:6790/0 conn(0x7f8930072620 legacy=0x7f8930151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:04.838 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.853+0000 7f8935c25640 1 -- 172.21.15.79:0/2928720832 shutdown_connections 2024-03-29T10:56:04.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.853+0000 7f8935c25640 1 -- 172.21.15.79:0/2928720832 >> 172.21.15.79:0/2928720832 conn(0x7f89300b3f40 msgr2=0x7f89300b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:04.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.854+0000 7f8935c25640 1 -- 172.21.15.79:0/2928720832 shutdown_connections 2024-03-29T10:56:04.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.854+0000 7f8935c25640 1 -- 172.21.15.79:0/2928720832 wait complete. 2024-03-29T10:56:04.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.854+0000 7f8935c25640 10 monclient: build_initial_monmap 2024-03-29T10:56:04.839 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.854+0000 7f8935c25640 10 monclient: monmap: 2024-03-29T10:56:04.839 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 1 librados: starting msgr at 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 1 librados: starting objecter 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 1 Processor -- start 2024-03-29T10:56:04.840 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 1 -- start start 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 1 librados: setting wanted keys 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 1 librados: calling monclient init 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.855+0000 7f8935c25640 10 monclient: init 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient(hunting): picked mon.noname-a con 0x7f8930072620 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient(hunting): picked mon.noname-c con 0x7f8930074650 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient(hunting): picked mon.noname-b con 0x7f8930144160 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300a9580 con 0x7f8930072620 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300aac00 con 0x7f8930144160 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8930146760 con 0x7f8930074650 2024-03-29T10:56:04.841 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:04.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.856+0000 7f8935c25640 10 monclient(hunting): authenticate will time out at 1484.148620s 2024-03-29T10:56:04.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.857+0000 7f892f7fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f8930074650 0x7f89301de690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:40978/0 (socket says 172.21.15.79:40978) 2024-03-29T10:56:04.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.857+0000 7f892f7fe640 1 -- 172.21.15.79:0/2945177490 learned_addr learned my addr 172.21.15.79:0/2945177490 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:04.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.857+0000 7f892cff9640 10 client.?.objecter ms_handle_connect 0x7f8930074650 2024-03-29T10:56:04.842 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.857+0000 7f892cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.859+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794634472 0 0) 0x7f8930146760 con 0x7f8930074650 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.859+0000 7f892cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.859+0000 7f892cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.859+0000 7f892cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.859+0000 7f892cff9640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.859+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910004eb0 con 0x7f8930074650 2024-03-29T10:56:04.849 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 209209370 0 0) 0x7f8910004eb0 con 0x7f8930074650 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8930146760 con 0x7f8930074650 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f89180017d0 con 0x7f8930074650 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 10 monclient(hunting): dump: 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.850 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.860+0000 7f892cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.861+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224105299 0 0) 0x7f8930146760 con 0x7f8930074650 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.861+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 >> v1:172.21.15.82:6789/0 conn(0x7f8930144160 legacy=0x7f89301deda0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.861+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 >> v1:172.21.15.79:6789/0 conn(0x7f8930072620 legacy=0x7f893013d910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 1 monclient: found mon.c 2024-03-29T10:56:04.851 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89301468f0 con 0x7f8930074650 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f892cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.862+0000 7f8935c25640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 10 monclient: _renew_subs 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893013e020 con 0x7f8930074650 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:04.852 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8918001930 con 0x7f8930074650 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8918004a00 con 0x7f8930074650 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892cff9640 10 monclient: dump: 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:04.853 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f892dffb640 4 set_mon_vals no callback set 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 10 monclient: _renew_subs 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930147230 con 0x7f8930074650 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.863+0000 7f8935c25640 1 librados: init done 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.865+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f891801a9d0 con 0x7f8930074650 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.865+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f8918084aa0 con 0x7f8930074650 2024-03-29T10:56:04.854 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.865+0000 7f892cff9640 10 client.4592.objecter ms_dispatch 0x7f8930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.865+0000 7f892cff9640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.865+0000 7f892cff9640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.865+0000 7f892cff9640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.866+0000 7f89277fe640 10 monclient: get_auth_request con 0x7f891006dce0 auth_method 0 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.867+0000 7f892cff9640 10 client.4592.objecter ms_handle_connect 0x7f891006dce0 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.867+0000 7f8935c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.867+0000 7f8935c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.867+0000 7f8935c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:04.855 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.867+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89300a9780 con 0x7f8930074650 2024-03-29T10:56:04.857 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.876+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89300a9780 con 0x7f8930074650 2024-03-29T10:56:04.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.876+0000 7f892cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:04.858 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:04.876+0000 7f892cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:05.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.204+0000 7f8935c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:05.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.204+0000 7f8935c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:05.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.204+0000 7f8935c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:05.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.204+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89300a9780 con 0x7f8930074650 2024-03-29T10:56:05.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.206+0000 7f892cff9640 1 -- 172.21.15.79:0/2945177490 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f89300a9780 con 0x7f8930074650 2024-03-29T10:56:05.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.206+0000 7f892cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:05.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.206+0000 7f892cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:05.187 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:05.187 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:05.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.209+0000 7f8935c25640 10 librados: watch_flush enter 2024-03-29T10:56:05.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 10 librados: watch_flush exit 2024-03-29T10:56:05.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 10 client.4592.objecter successfully canceled tick 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 >> v1:172.21.15.79:6816/4290694627 conn(0x7f891006dce0 legacy=0x7f89100701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 10 monclient: shutdown 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 >> v1:172.21.15.79:6790/0 conn(0x7f8930074650 legacy=0x7f89301de690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 shutdown_connections 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.210+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 >> 172.21.15.79:0/2945177490 conn(0x7f89300b3f40 msgr2=0x7f89300b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.211+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 shutdown_connections 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.211+0000 7f8935c25640 1 -- 172.21.15.79:0/2945177490 wait complete. 2024-03-29T10:56:05.192 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:05.211+0000 7f8935c25640 1 librados: shutdown 2024-03-29T10:56:05.208 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:12.014 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.195+0000 7fdfc208f640 10 monclient: get_monmap_and_config 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.195+0000 7fdfc208f640 10 monclient: build_initial_monmap 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.195+0000 7fdfc208f640 10 monclient: monmap: 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:12.196020+0000 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:12.196020+0000 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:12.176 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:12.177 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.196+0000 7fdfc208f640 1 Processor -- start 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.196+0000 7fdfc208f640 1 -- start start 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.196+0000 7fdfc208f640 10 monclient: init 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient(hunting): picked mon.noname-c con 0x7fdfbc0b0860 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient(hunting): picked mon.noname-a con 0x7fdfbc13ef30 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient(hunting): picked mon.noname-b con 0x7fdfbc0af8e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc0b0c40 con 0x7fdfbc13ef30 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc0aed10 con 0x7fdfbc0b0860 2024-03-29T10:56:12.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc0afd50 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:12.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.197+0000 7fdfc208f640 10 monclient(hunting): authenticate will time out at 1491.489420s 2024-03-29T10:56:12.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.199+0000 7fdfbbfff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fdfbc0af8e0 0x7fdfbc149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:52382/0 (socket says 172.21.15.79:52382) 2024-03-29T10:56:12.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.199+0000 7fdfbbfff640 1 -- 172.21.15.79:0/3685909363 learned_addr learned my addr 172.21.15.79:0/3685909363 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:12.181 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.200+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288296695 0 0) 0x7fdfbc0afd50 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.181 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.200+0000 7fdfba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:12.181 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.200+0000 7fdfba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:12.181 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.200+0000 7fdfba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:12.181 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.200+0000 7fdfba7fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:56:12.181 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.200+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa4003520 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.202+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035971659 0 0) 0x7fdfbc0b0c40 con 0x7fdfbc13ef30 2024-03-29T10:56:12.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.202+0000 7fdfba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:12.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.202+0000 7fdfba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:12.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.202+0000 7fdfba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:12.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.202+0000 7fdfba7fc640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:56:12.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.202+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfbc0afd50 con 0x7fdfbc13ef30 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4220251146 0 0) 0x7fdfa4003520 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 >> v1:172.21.15.82:6789/0 conn(0x7fdfbc0b0860 legacy=0x7fdfbc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 >> v1:172.21.15.79:6789/0 conn(0x7fdfbc13ef30 legacy=0x7fdfbc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 monclient: found mon.noname-b 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc0ab1b0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:12.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdfb0004cb0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: dump: 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:12.185 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:12.186 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfb0004e90 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 1 -- 172.21.15.79:0/3685909363 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdfb00013f0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.203+0000 7fdfba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.204+0000 7fdfba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.204+0000 7fdfba7fc640 10 monclient: dump: 2024-03-29T10:56:12.187 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.204+0000 7fdfc208f640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.204+0000 7fdfc208f640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.204+0000 7fdfc208f640 4 set_mon_vals no callback set 2024-03-29T10:56:12.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.205+0000 7fdfc208f640 10 monclient: shutdown 2024-03-29T10:56:12.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.205+0000 7fdfc208f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:12.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.205+0000 7fdfc208f640 1 -- 172.21.15.79:0/3685909363 >> v1:172.21.15.79:6790/0 conn(0x7fdfbc0af8e0 legacy=0x7fdfbc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:12.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.205+0000 7fdfc208f640 1 -- 172.21.15.79:0/3685909363 shutdown_connections 2024-03-29T10:56:12.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.205+0000 7fdfc208f640 1 -- 172.21.15.79:0/3685909363 >> 172.21.15.79:0/3685909363 conn(0x7fdfbc06b660 msgr2=0x7fdfbc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:12.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.208+0000 7fdfc208f640 1 -- 172.21.15.79:0/3685909363 shutdown_connections 2024-03-29T10:56:12.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.209+0000 7fdfc208f640 1 -- 172.21.15.79:0/3685909363 wait complete. 2024-03-29T10:56:12.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 10 monclient: build_initial_monmap 2024-03-29T10:56:12.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 10 monclient: monmap: 2024-03-29T10:56:12.190 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:12.190 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:12.191 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:12.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 1 librados: starting msgr at 2024-03-29T10:56:12.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 1 librados: starting objecter 2024-03-29T10:56:12.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 1 Processor -- start 2024-03-29T10:56:12.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 1 -- start start 2024-03-29T10:56:12.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 1 librados: setting wanted keys 2024-03-29T10:56:12.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 1 librados: calling monclient init 2024-03-29T10:56:12.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.210+0000 7fdfc208f640 10 monclient: init 2024-03-29T10:56:12.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:12.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:12.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient(hunting): picked mon.noname-c con 0x7fdfbc0af8e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:12.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient(hunting): picked mon.noname-b con 0x7fdfbc0b0860 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:12.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient(hunting): picked mon.noname-a con 0x7fdfbc13ef30 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:12.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc1c85f0 con 0x7fdfbc13ef30 2024-03-29T10:56:12.201 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc1c7000 con 0x7fdfbc0b0860 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc1c8840 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.211+0000 7fdfc208f640 10 monclient(hunting): authenticate will time out at 1491.504020s 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.212+0000 7fdfbb7fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fdfbc0af8e0 0x7fdfbc1c7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:60240/0 (socket says 172.21.15.79:60240) 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.212+0000 7fdfbb7fe640 1 -- 172.21.15.79:0/3220465565 learned_addr learned my addr 172.21.15.79:0/3220465565 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.213+0000 7fdfa3fff640 10 client.?.objecter ms_handle_connect 0x7fdfbc0af8e0 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.213+0000 7fdfa3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.215+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860385750 0 0) 0x7fdfbc1c8840 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.215+0000 7fdfa3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.215+0000 7fdfa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:12.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.215+0000 7fdfa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.215+0000 7fdfa3fff640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.215+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa8004eb0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673531761 0 0) 0x7fdfa8004eb0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfbc1c8840 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdfac004080 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): dump: 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:12.203 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 client.?.objecter ms_handle_connect 0x7fdfbc0b0860 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:12.204 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297590598 0 0) 0x7fdfbc1c7000 con 0x7fdfbc0b0860 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa8004eb0 con 0x7fdfbc0b0860 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.216+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1635657156 0 0) 0x7fdfbc1c8840 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.217+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 >> v1:172.21.15.82:6789/0 conn(0x7fdfbc0b0860 legacy=0x7fdfbc139350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.217+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 >> v1:172.21.15.79:6789/0 conn(0x7fdfbc13ef30 legacy=0x7fdfbc1346a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 1 monclient: found mon.c 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc1c8b20 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 10 monclient: _finish_auth 0 2024-03-29T10:56:12.205 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.218+0000 7fdfa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 10 monclient: _renew_subs 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfbc1c68c0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 10 monclient: _renew_subs 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfbc134db0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.206 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.219+0000 7fdfc208f640 1 librados: init done 2024-03-29T10:56:12.209 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.220+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfac0044d0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.209 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.220+0000 7fdfa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.220+0000 7fdfb9ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfc208f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdfac0042e0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfa3fff640 10 monclient: dump: 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:12.210 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfc208f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfc208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.221+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c005080 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.222+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fdfac0013e0 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.222+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fdfac084b70 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.223+0000 7fdfa3fff640 10 client.4598.objecter ms_dispatch 0x7fdfbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.223+0000 7fdfa3fff640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.223+0000 7fdfa3fff640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:12.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.223+0000 7fdfa3fff640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T10:56:12.212 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.223+0000 7fdfbaffd640 10 monclient: get_auth_request con 0x7fdfa806dd20 auth_method 0 2024-03-29T10:56:12.212 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.223+0000 7fdfa3fff640 10 client.4598.objecter ms_handle_connect 0x7fdfa806dd20 2024-03-29T10:56:12.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.232+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfac050450 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.232+0000 7fdfa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:12.213 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.232+0000 7fdfa3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:12.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.570+0000 7fdfc208f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:12.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.570+0000 7fdfc208f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:12.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.570+0000 7fdfc208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:12.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.570+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf9c005330 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.571+0000 7fdfa3fff640 1 -- 172.21.15.79:0/3220465565 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fdfac052380 con 0x7fdfbc0af8e0 2024-03-29T10:56:12.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.571+0000 7fdfa3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:12.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.571+0000 7fdfa3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:12.552 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:12.553 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:12.555 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.574+0000 7fdfc208f640 10 librados: watch_flush enter 2024-03-29T10:56:12.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.575+0000 7fdfc208f640 10 librados: watch_flush exit 2024-03-29T10:56:12.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.575+0000 7fdfc208f640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T10:56:12.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.575+0000 7fdfc208f640 10 client.4598.objecter successfully canceled tick 2024-03-29T10:56:12.558 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.577+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 >> v1:172.21.15.79:6816/4290694627 conn(0x7fdfa806dd20 legacy=0x7fdfa80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:12.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.578+0000 7fdfc208f640 10 monclient: shutdown 2024-03-29T10:56:12.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.578+0000 7fdfc208f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:12.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.578+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 >> v1:172.21.15.79:6790/0 conn(0x7fdfbc0af8e0 legacy=0x7fdfbc1c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:12.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.579+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 shutdown_connections 2024-03-29T10:56:12.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.579+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 >> 172.21.15.79:0/3220465565 conn(0x7fdfbc06b660 msgr2=0x7fdfbc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:12.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.580+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 shutdown_connections 2024-03-29T10:56:12.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.580+0000 7fdfc208f640 1 -- 172.21.15.79:0/3220465565 wait complete. 2024-03-29T10:56:12.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:12.580+0000 7fdfc208f640 1 librados: shutdown 2024-03-29T10:56:12.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:19.384 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:19.546 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.565+0000 7f669f1e4640 10 monclient: get_monmap_and_config 2024-03-29T10:56:19.546 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.565+0000 7f669f1e4640 10 monclient: build_initial_monmap 2024-03-29T10:56:19.546 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.565+0000 7f669f1e4640 10 monclient: monmap: 2024-03-29T10:56:19.546 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:19.546 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:19.566650+0000 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:19.566650+0000 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:19.547 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.567+0000 7f669f1e4640 1 Processor -- start 2024-03-29T10:56:19.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.567+0000 7f669f1e4640 1 -- start start 2024-03-29T10:56:19.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.567+0000 7f669f1e4640 10 monclient: init 2024-03-29T10:56:19.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.567+0000 7f669f1e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 10 monclient(hunting): picked mon.noname-b con 0x7f66980b1f60 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 10 monclient(hunting): picked mon.noname-c con 0x7f6698071eb0 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 10 monclient(hunting): picked mon.noname-a con 0x7f6698143d10 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669806ef80 con 0x7f6698143d10 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669806f260 con 0x7f6698071eb0 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980b40c0 con 0x7f66980b1f60 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:19.549 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.568+0000 7f669f1e4640 10 monclient(hunting): authenticate will time out at 1498.860756s 2024-03-29T10:56:19.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.571+0000 7f669d781640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f6698143d10 0x7f6698146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:40419/0 (socket says 172.21.15.79:40419) 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.571+0000 7f669d781640 1 -- 172.21.15.79:0/1228229796 learned_addr learned my addr 172.21.15.79:0/1228229796 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.572+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321214830 0 0) 0x7f669806ef80 con 0x7f6698143d10 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.572+0000 7f66977fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.572+0000 7f66977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.572+0000 7f66977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.572+0000 7f66977fe640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.572+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6680003520 con 0x7f6698143d10 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.573+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2014729530 0 0) 0x7f6680003520 con 0x7f6698143d10 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.573+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 >> v1:172.21.15.79:6790/0 conn(0x7f66980b1f60 legacy=0x7f6698143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:19.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.573+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 >> v1:172.21.15.82:6789/0 conn(0x7f6698071eb0 legacy=0x7f6698146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 1 monclient: found mon.noname-a 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698155d70 con 0x7f6698143d10 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6690001a10 con 0x7f6698143d10 2024-03-29T10:56:19.556 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f66977fe640 10 monclient: dump: 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:19.557 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f669f1e4640 5 monclient: authenticate success, global_id 4518 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.574+0000 7f669f1e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6690001be0 con 0x7f6698143d10 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f66977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f66977fe640 1 -- 172.21.15.79:0/1228229796 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6690002030 con 0x7f6698143d10 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f66977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f66977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f66977fe640 10 monclient: dump: 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:19.560 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f669f1e4640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.575+0000 7f669f1e4640 4 set_mon_vals no callback set 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.576+0000 7f669f1e4640 10 monclient: shutdown 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.576+0000 7f669f1e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.576+0000 7f669f1e4640 1 -- 172.21.15.79:0/1228229796 >> v1:172.21.15.79:6789/0 conn(0x7f6698143d10 legacy=0x7f6698146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.576+0000 7f669f1e4640 1 -- 172.21.15.79:0/1228229796 shutdown_connections 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.576+0000 7f669f1e4640 1 -- 172.21.15.79:0/1228229796 >> 172.21.15.79:0/1228229796 conn(0x7f6698066a00 msgr2=0x7f6698066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:19.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.577+0000 7f669f1e4640 1 -- 172.21.15.79:0/1228229796 shutdown_connections 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.577+0000 7f669f1e4640 1 -- 172.21.15.79:0/1228229796 wait complete. 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 10 monclient: build_initial_monmap 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 10 monclient: monmap: 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 1 librados: starting msgr at 2024-03-29T10:56:19.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 1 librados: starting objecter 2024-03-29T10:56:19.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 1 Processor -- start 2024-03-29T10:56:19.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 1 -- start start 2024-03-29T10:56:19.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 1 librados: setting wanted keys 2024-03-29T10:56:19.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 1 librados: calling monclient init 2024-03-29T10:56:19.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.578+0000 7f669f1e4640 10 monclient: init 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient(hunting): picked mon.noname-c con 0x7f66980b1f60 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient(hunting): picked mon.noname-b con 0x7f6698143d10 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient(hunting): picked mon.noname-a con 0x7f66980ab7e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698140720 con 0x7f66980ab7e0 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980ac820 con 0x7f6698143d10 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980acb00 con 0x7f66980b1f60 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.580+0000 7f669f1e4640 10 monclient(hunting): authenticate will time out at 1498.872303s 2024-03-29T10:56:19.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.584+0000 7f669d781640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f66980ab7e0 0x7f6698140010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:41637/0 (socket says 172.21.15.79:41637) 2024-03-29T10:56:19.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.584+0000 7f669d781640 1 -- 172.21.15.79:0/434839677 learned_addr learned my addr 172.21.15.79:0/434839677 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:19.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 client.?.objecter ms_handle_connect 0x7f6698143d10 2024-03-29T10:56:19.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:19.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 client.?.objecter ms_handle_connect 0x7f66980b1f60 2024-03-29T10:56:19.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 client.?.objecter ms_handle_connect 0x7f66980ab7e0 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839733898 0 0) 0x7f6698140720 con 0x7f66980ab7e0 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6684004eb0 con 0x7f66980ab7e0 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988050305 0 0) 0x7f66980acb00 con 0x7f66980b1f60 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:56:19.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6698140720 con 0x7f66980b1f60 2024-03-29T10:56:19.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568478022 0 0) 0x7f66980ac820 con 0x7f6698143d10 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66980acb00 con 0x7f6698143d10 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040873693 0 0) 0x7f66980acb00 con 0x7f6698143d10 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66980ac820 con 0x7f6698143d10 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6688001730 con 0x7f6698143d10 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): dump: 2024-03-29T10:56:19.570 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 999026096 0 0) 0x7f6698140720 con 0x7f66980b1f60 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66980acb00 con 0x7f66980b1f60 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f668c001490 con 0x7f66980b1f60 2024-03-29T10:56:19.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.586+0000 7f66957fa640 10 monclient(hunting): dump: 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 511512639 0 0) 0x7f66980ac820 con 0x7f6698143d10 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 >> v1:172.21.15.79:6790/0 conn(0x7f66980b1f60 legacy=0x7f66981e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:19.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 >> v1:172.21.15.79:6789/0 conn(0x7f66980ab7e0 legacy=0x7f6698140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 monclient: found mon.b 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66981e75a0 con 0x7f6698143d10 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6688001c40 con 0x7f6698143d10 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f6688003830 con 0x7f6698143d10 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f66957fa640 10 monclient: dump: 2024-03-29T10:56:19.573 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f6696ffd640 4 set_mon_vals no callback set 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 5 monclient: authenticate success, global_id 4471 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 10 monclient: _renew_subs 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:19.574 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66981e7230 con 0x7f6698143d10 2024-03-29T10:56:19.575 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:19.575 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 10 monclient: _renew_subs 2024-03-29T10:56:19.575 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:19.575 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66980ab420 con 0x7f6698143d10 2024-03-29T10:56:19.575 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.588+0000 7f669f1e4640 1 librados: init done 2024-03-29T10:56:19.577 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.590+0000 7f669f1e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:19.577 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.590+0000 7f669f1e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.590+0000 7f669f1e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.590+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66981e7790 con 0x7f6698143d10 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.591+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f6688019800 con 0x7f6698143d10 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.592+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f6688083680 con 0x7f6698143d10 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.592+0000 7f66957fa640 10 client.4471.objecter ms_dispatch 0x7f6698000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.592+0000 7f66957fa640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.592+0000 7f66957fa640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.592+0000 7f669cf80640 10 monclient: get_auth_request con 0x7f668406def0 auth_method 0 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.593+0000 7f66957fa640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-29T10:56:19.578 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.593+0000 7f66957fa640 10 client.4471.objecter ms_handle_connect 0x7f668406def0 2024-03-29T10:56:19.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.598+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f668804ef60 con 0x7f6698143d10 2024-03-29T10:56:19.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.598+0000 7f66957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:19.579 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.598+0000 7f66957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:19.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.973+0000 7f669f1e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:19.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.973+0000 7f669f1e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:19.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.973+0000 7f669f1e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:19.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.973+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66981e73c0 con 0x7f6698143d10 2024-03-29T10:56:19.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.974+0000 7f66957fa640 1 -- 172.21.15.79:0/434839677 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f6688050e90 con 0x7f6698143d10 2024-03-29T10:56:19.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.974+0000 7f66957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:19.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.974+0000 7f66957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:19.955 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:19.955 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:19.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.978+0000 7f669f1e4640 10 librados: watch_flush enter 2024-03-29T10:56:19.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.978+0000 7f669f1e4640 10 librados: watch_flush exit 2024-03-29T10:56:19.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.978+0000 7f669f1e4640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-29T10:56:19.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.978+0000 7f669f1e4640 10 client.4471.objecter successfully canceled tick 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 >> v1:172.21.15.79:6816/4290694627 conn(0x7f668406def0 legacy=0x7f66840703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 10 monclient: shutdown 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 >> v1:172.21.15.82:6789/0 conn(0x7f6698143d10 legacy=0x7f669813f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 shutdown_connections 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 >> 172.21.15.79:0/434839677 conn(0x7f6698066a00 msgr2=0x7f6698146960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 shutdown_connections 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 -- 172.21.15.79:0/434839677 wait complete. 2024-03-29T10:56:19.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:19.979+0000 7f669f1e4640 1 librados: shutdown 2024-03-29T10:56:19.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:26.788 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:26.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.963+0000 7fa0f69cf640 10 monclient: get_monmap_and_config 2024-03-29T10:56:26.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.963+0000 7fa0f69cf640 10 monclient: build_initial_monmap 2024-03-29T10:56:26.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.963+0000 7fa0f69cf640 10 monclient: monmap: 2024-03-29T10:56:26.944 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:26.944 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:26.944 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:26.964722+0000 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:26.964722+0000 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:26.945 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.965+0000 7fa0f69cf640 1 Processor -- start 2024-03-29T10:56:26.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.965+0000 7fa0f69cf640 1 -- start start 2024-03-29T10:56:26.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.965+0000 7fa0f69cf640 10 monclient: init 2024-03-29T10:56:26.946 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.965+0000 7fa0f69cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f01457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f013f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f00a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f00aaef0 con 0x7fa0f01457f0 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f0145bd0 con 0x7fa0f013f650 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f0145eb0 con 0x7fa0f00a9d70 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0f69cf640 10 monclient(hunting): authenticate will time out at 1506.258370s 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0effff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fa0f01457f0 0x7fa0f013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:46024/0 (socket says 172.21.15.79:46024) 2024-03-29T10:56:26.947 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0effff640 1 -- 172.21.15.79:0/3793439494 learned_addr learned my addr 172.21.15.79:0/3793439494 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677189777 0 0) 0x7fa0f0145eb0 con 0x7fa0f00a9d70 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d8003520 con 0x7fa0f00a9d70 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201761787 0 0) 0x7fa0f00aaef0 con 0x7fa0f01457f0 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:56:26.948 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.966+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f0145eb0 con 0x7fa0f01457f0 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888475983 0 0) 0x7fa0d8003520 con 0x7fa0f00a9d70 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 >> v1:172.21.15.82:6789/0 conn(0x7fa0f013f650 legacy=0x7fa0f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 >> v1:172.21.15.79:6789/0 conn(0x7fa0f01457f0 legacy=0x7fa0f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0146340 con 0x7fa0f00a9d70 2024-03-29T10:56:26.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.967+0000 7fa0eeffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0f69cf640 5 monclient: authenticate success, global_id 4604 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa0e8001a10 con 0x7fa0f00a9d70 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0f69cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: dump: 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:26.951 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0f69cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e8001b70 con 0x7fa0f00a9d70 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 1 -- 172.21.15.79:0/3793439494 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa0e8001fc0 con 0x7fa0f00a9d70 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0eeffd640 10 monclient: dump: 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.952 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0f69cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.968+0000 7fa0f69cf640 4 set_mon_vals no callback set 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 10 monclient: shutdown 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 1 -- 172.21.15.79:0/3793439494 >> v1:172.21.15.79:6790/0 conn(0x7fa0f00a9d70 legacy=0x7fa0f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 1 -- 172.21.15.79:0/3793439494 shutdown_connections 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 1 -- 172.21.15.79:0/3793439494 >> 172.21.15.79:0/3793439494 conn(0x7fa0f0066a00 msgr2=0x7fa0f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 1 -- 172.21.15.79:0/3793439494 shutdown_connections 2024-03-29T10:56:26.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.969+0000 7fa0f69cf640 1 -- 172.21.15.79:0/3793439494 wait complete. 2024-03-29T10:56:26.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.970+0000 7fa0f69cf640 10 monclient: build_initial_monmap 2024-03-29T10:56:26.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.970+0000 7fa0f69cf640 10 monclient: monmap: 2024-03-29T10:56:26.958 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 1 librados: starting msgr at 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 1 librados: starting objecter 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 1 Processor -- start 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 1 -- start start 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 1 librados: setting wanted keys 2024-03-29T10:56:26.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 1 librados: calling monclient init 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.971+0000 7fa0f69cf640 10 monclient: init 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f00a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f013f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f01457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f0070c90 con 0x7fa0f01457f0 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f01dfa50 con 0x7fa0f00a9d70 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f01dbbf0 con 0x7fa0f013f650 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0f69cf640 10 monclient(hunting): authenticate will time out at 1506.264588s 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0effff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fa0f00a9d70 0x7fa0f01dec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:34110/0 (socket says 172.21.15.79:34110) 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0effff640 1 -- 172.21.15.79:0/2150809383 learned_addr learned my addr 172.21.15.79:0/2150809383 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:26.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0ecff9640 10 client.?.objecter ms_handle_connect 0x7fa0f013f650 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0ecff9640 10 client.?.objecter ms_handle_connect 0x7fa0f00a9d70 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0ecff9640 10 client.?.objecter ms_handle_connect 0x7fa0f01457f0 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.972+0000 7fa0ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007651301 0 0) 0x7fa0f01dfa50 con 0x7fa0f00a9d70 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f00a9d70 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938258718 0 0) 0x7fa0f01dbbf0 con 0x7fa0f013f650 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:26.961 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f01dfa50 con 0x7fa0f013f650 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562161097 0 0) 0x7fa0f0070c90 con 0x7fa0f01457f0 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f01dbbf0 con 0x7fa0f01457f0 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326913311 0 0) 0x7fa0dc004eb0 con 0x7fa0f00a9d70 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f0070c90 con 0x7fa0f00a9d70 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa0e0001930 con 0x7fa0f00a9d70 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:26.962 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): dump: 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719261776 0 0) 0x7fa0f01dbbf0 con 0x7fa0f01457f0 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f01457f0 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa0e8001e80 con 0x7fa0f01457f0 2024-03-29T10:56:26.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.973+0000 7fa0ecff9640 10 monclient(hunting): dump: 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2317697994 0 0) 0x7fa0f0070c90 con 0x7fa0f00a9d70 2024-03-29T10:56:26.964 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 >> v1:172.21.15.79:6790/0 conn(0x7fa0f013f650 legacy=0x7fa0f006fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 >> v1:172.21.15.79:6789/0 conn(0x7fa0f01457f0 legacy=0x7fa0f01df340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 monclient: found mon.b 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0070ec0 con 0x7fa0f00a9d70 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 10 monclient: _renew_subs 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f0071200 con 0x7fa0f00a9d70 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:26.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 10 monclient: _renew_subs 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f006e190 con 0x7fa0f00a9d70 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0f69cf640 1 librados: init done 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e0001a90 con 0x7fa0f00a9d70 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa0e0004a90 con 0x7fa0f00a9d70 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.974+0000 7fa0ecff9640 10 monclient: dump: 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:26.966 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.975+0000 7fa0ee7fc640 4 set_mon_vals no callback set 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.976+0000 7fa0f69cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.976+0000 7fa0f69cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.976+0000 7fa0f69cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:26.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.976+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d0005080 con 0x7fa0f00a9d70 2024-03-29T10:56:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.978+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fa0e001aa60 con 0x7fa0f00a9d70 2024-03-29T10:56:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ef7fe640 10 monclient: get_auth_request con 0x7fa0dc06dfb0 auth_method 0 2024-03-29T10:56:26.968 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fa0e0084a80 con 0x7fa0f00a9d70 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ecff9640 10 client.4474.objecter ms_dispatch 0x7fa0f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ecff9640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ecff9640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ecff9640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.979+0000 7fa0ecff9640 10 client.4474.objecter ms_handle_connect 0x7fa0dc06dfb0 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.986+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0e0050190 con 0x7fa0f00a9d70 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.987+0000 7fa0ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:26.969 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:26.987+0000 7fa0ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:27.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.341+0000 7fa0f69cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:27.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.341+0000 7fa0f69cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:27.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.341+0000 7fa0f69cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:56:27.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.341+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0d0005330 con 0x7fa0f00a9d70 2024-03-29T10:56:27.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.342+0000 7fa0ecff9640 1 -- 172.21.15.79:0/2150809383 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fa0e0052290 con 0x7fa0f00a9d70 2024-03-29T10:56:27.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.342+0000 7fa0ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:27.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.342+0000 7fa0ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:27.324 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:27.324 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:27.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 10 librados: watch_flush enter 2024-03-29T10:56:27.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 10 librados: watch_flush exit 2024-03-29T10:56:27.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:56:27.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:56:27.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 >> v1:172.21.15.79:6816/4290694627 conn(0x7fa0dc06dfb0 legacy=0x7fa0dc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:27.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 10 monclient: shutdown 2024-03-29T10:56:27.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:27.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.346+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 >> v1:172.21.15.82:6789/0 conn(0x7fa0f00a9d70 legacy=0x7fa0f01dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:27.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.348+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 shutdown_connections 2024-03-29T10:56:27.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.348+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 >> 172.21.15.79:0/2150809383 conn(0x7fa0f0066a00 msgr2=0x7fa0f013ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:27.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.348+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 shutdown_connections 2024-03-29T10:56:27.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.348+0000 7fa0f69cf640 1 -- 172.21.15.79:0/2150809383 wait complete. 2024-03-29T10:56:27.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:27.350+0000 7fa0f69cf640 1 librados: shutdown 2024-03-29T10:56:27.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:34.154 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:34.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.325+0000 7fa5457cf640 10 monclient: get_monmap_and_config 2024-03-29T10:56:34.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.325+0000 7fa5457cf640 10 monclient: build_initial_monmap 2024-03-29T10:56:34.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.325+0000 7fa5457cf640 10 monclient: monmap: 2024-03-29T10:56:34.308 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:34.308 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.309 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:34.326860+0000 2024-03-29T10:56:34.309 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:34.326860+0000 2024-03-29T10:56:34.309 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:34.309 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.309 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:34.310 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:34.310 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:34.310 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.326+0000 7fa5457cf640 1 Processor -- start 2024-03-29T10:56:34.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 1 -- start start 2024-03-29T10:56:34.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient: init 2024-03-29T10:56:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa5401422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa540142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa5400b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:34.321 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540142c80 con 0x7fa540142690 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400aae40 con 0x7fa5401422b0 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54013fbd0 con 0x7fa5400b17d0 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.327+0000 7fa5457cf640 10 monclient(hunting): authenticate will time out at 1513.620009s 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53e7fc640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7fa540142690 0x7fa5400b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:40504/0 (socket says 172.21.15.79:40504) 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53e7fc640 1 -- 172.21.15.79:0/3206577473 learned_addr learned my addr 172.21.15.79:0/3206577473 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567607117 0 0) 0x7fa540142c80 con 0x7fa540142690 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53dffb640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.328+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa52c003520 con 0x7fa540142690 2024-03-29T10:56:34.322 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332760151 0 0) 0x7fa54013fbd0 con 0x7fa5400b17d0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540142c80 con 0x7fa5400b17d0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835809628 0 0) 0x7fa540142c80 con 0x7fa5400b17d0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 >> v1:172.21.15.82:6789/0 conn(0x7fa5401422b0 legacy=0x7fa54013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 >> v1:172.21.15.79:6789/0 conn(0x7fa540142690 legacy=0x7fa5400b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 monclient: found mon.noname-b 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540151940 con 0x7fa5400b17d0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:34.323 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa530001670 con 0x7fa5400b17d0 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa53dffb640 10 monclient: dump: 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:34.324 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa5457cf640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.329+0000 7fa5457cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.330+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5300017d0 con 0x7fa5400b17d0 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.330+0000 7fa53dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.330+0000 7fa53dffb640 1 -- 172.21.15.79:0/3206577473 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa530001ba0 con 0x7fa5400b17d0 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.330+0000 7fa5457cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.330+0000 7fa5457cf640 4 set_mon_vals no callback set 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa53dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa53dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa53dffb640 10 monclient: dump: 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.325 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa5457cf640 10 monclient: shutdown 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa5457cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa5457cf640 1 -- 172.21.15.79:0/3206577473 >> v1:172.21.15.79:6790/0 conn(0x7fa5400b17d0 legacy=0x7fa5400b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa5457cf640 1 -- 172.21.15.79:0/3206577473 shutdown_connections 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa5457cf640 1 -- 172.21.15.79:0/3206577473 >> 172.21.15.79:0/3206577473 conn(0x7fa540066a00 msgr2=0x7fa540066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.331+0000 7fa5457cf640 1 -- 172.21.15.79:0/3206577473 shutdown_connections 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.332+0000 7fa5457cf640 1 -- 172.21.15.79:0/3206577473 wait complete. 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.332+0000 7fa5457cf640 10 monclient: build_initial_monmap 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.332+0000 7fa5457cf640 10 monclient: monmap: 2024-03-29T10:56:34.326 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 1 librados: starting msgr at 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 1 librados: starting objecter 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 1 Processor -- start 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 1 -- start start 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 1 librados: setting wanted keys 2024-03-29T10:56:34.327 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 1 librados: calling monclient init 2024-03-29T10:56:34.328 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.333+0000 7fa5457cf640 10 monclient: init 2024-03-29T10:56:34.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:34.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:34.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa5400b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:34.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa5401422b0 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:34.329 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa540142690 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54012fba0 con 0x7fa5401422b0 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400acc30 con 0x7fa540142690 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540132a30 con 0x7fa5400b17d0 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5457cf640 10 monclient(hunting): authenticate will time out at 1513.626878s 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa53effd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fa5400b17d0 0x7fa5400b0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:45176/0 (socket says 172.21.15.79:45176) 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa53effd640 1 -- 172.21.15.79:0/634410965 learned_addr learned my addr 172.21.15.79:0/634410965 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5377fe640 10 client.?.objecter ms_handle_connect 0x7fa5400b17d0 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.334+0000 7fa5377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 client.?.objecter ms_handle_connect 0x7fa540142690 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:34.330 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033046191 0 0) 0x7fa540132a30 con 0x7fa5400b17d0 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa520004eb0 con 0x7fa5400b17d0 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 client.?.objecter ms_handle_connect 0x7fa5401422b0 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626190911 0 0) 0x7fa5400acc30 con 0x7fa540142690 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.335+0000 7fa5377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540132a30 con 0x7fa540142690 2024-03-29T10:56:34.331 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64393581 0 0) 0x7fa54012fba0 con 0x7fa5401422b0 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5400acc30 con 0x7fa5401422b0 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898365968 0 0) 0x7fa520004eb0 con 0x7fa5400b17d0 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa54012fba0 con 0x7fa5400b17d0 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa5280019d0 con 0x7fa5400b17d0 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): dump: 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.332 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160265623 0 0) 0x7fa5400acc30 con 0x7fa5401422b0 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa520004eb0 con 0x7fa5401422b0 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.336+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17697653 0 0) 0x7fa540132a30 con 0x7fa540142690 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5400acc30 con 0x7fa540142690 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa538001bd0 con 0x7fa5401422b0 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:34.333 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient(hunting): dump: 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.334 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa530001ae0 con 0x7fa540142690 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient(hunting): dump: 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.335 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1424874536 0 0) 0x7fa54012fba0 con 0x7fa5400b17d0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 >> v1:172.21.15.82:6789/0 conn(0x7fa540142690 legacy=0x7fa5401e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 >> v1:172.21.15.79:6789/0 conn(0x7fa5401422b0 legacy=0x7fa5401dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 monclient: found mon.c 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400ace80 con 0x7fa5400b17d0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 5 monclient: authenticate success, global_id 4613 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 10 monclient: _renew_subs 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5400ad160 con 0x7fa5400b17d0 2024-03-29T10:56:34.336 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 10 monclient: _renew_subs 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5400abf20 con 0x7fa5400b17d0 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5457cf640 1 librados: init done 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa528001b30 con 0x7fa5400b17d0 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fa528003a60 con 0x7fa5400b17d0 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.337+0000 7fa5377fe640 10 monclient: dump: 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:34.337 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.338+0000 7fa53d7fa640 4 set_mon_vals no callback set 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.338+0000 7fa5457cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.339+0000 7fa5457cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.339+0000 7fa5457cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:34.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.339+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa524005080 con 0x7fa5400b17d0 2024-03-29T10:56:34.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.342+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fa528003bc0 con 0x7fa5400b17d0 2024-03-29T10:56:34.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fa5280849d0 con 0x7fa5400b17d0 2024-03-29T10:56:34.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa53e7fc640 10 monclient: get_auth_request con 0x7fa52006e120 auth_method 0 2024-03-29T10:56:34.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa5377fe640 10 client.4613.objecter ms_dispatch 0x7fa540000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:34.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa5377fe640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:34.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa5377fe640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:34.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa5377fe640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T10:56:34.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.343+0000 7fa5377fe640 10 client.4613.objecter ms_handle_connect 0x7fa52006e120 2024-03-29T10:56:34.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.354+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa528050310 con 0x7fa5400b17d0 2024-03-29T10:56:34.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.354+0000 7fa5377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:34.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.354+0000 7fa5377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:34.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.744+0000 7fa5457cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:34.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.744+0000 7fa5457cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:34.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.744+0000 7fa5457cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:34.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.744+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa524005330 con 0x7fa5400b17d0 2024-03-29T10:56:34.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.745+0000 7fa5377fe640 1 -- 172.21.15.79:0/634410965 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fa528052240 con 0x7fa5400b17d0 2024-03-29T10:56:34.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.745+0000 7fa5377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:34.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.745+0000 7fa5377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:34.726 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:34.726 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:34.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 10 librados: watch_flush enter 2024-03-29T10:56:34.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 10 librados: watch_flush exit 2024-03-29T10:56:34.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T10:56:34.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 10 client.4613.objecter successfully canceled tick 2024-03-29T10:56:34.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 >> v1:172.21.15.79:6816/4290694627 conn(0x7fa52006e120 legacy=0x7fa5200705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 10 monclient: shutdown 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 >> v1:172.21.15.79:6790/0 conn(0x7fa5400b17d0 legacy=0x7fa5400b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.749+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 shutdown_connections 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.750+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 >> 172.21.15.79:0/634410965 conn(0x7fa540066a00 msgr2=0x7fa54013d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.750+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 shutdown_connections 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.750+0000 7fa5457cf640 1 -- 172.21.15.79:0/634410965 wait complete. 2024-03-29T10:56:34.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:34.750+0000 7fa5457cf640 1 librados: shutdown 2024-03-29T10:56:34.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:41.553 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.723+0000 7f1c1e123640 10 monclient: get_monmap_and_config 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.723+0000 7f1c1e123640 10 monclient: build_initial_monmap 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.723+0000 7f1c1e123640 10 monclient: monmap: 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:41.725716+0000 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:41.725716+0000 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:41.706 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:41.707 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:41.707 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:41.707 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:41.707 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:41.707 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.725+0000 7f1c1e123640 1 Processor -- start 2024-03-29T10:56:41.708 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.725+0000 7f1c1e123640 1 -- start start 2024-03-29T10:56:41.708 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.725+0000 7f1c1e123640 10 monclient: init 2024-03-29T10:56:41.708 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.725+0000 7f1c1e123640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:41.714 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:41.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 10 monclient(hunting): picked mon.noname-c con 0x7f1c1806d9d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:41.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 10 monclient(hunting): picked mon.noname-b con 0x7f1c1813e820 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:41.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 10 monclient(hunting): picked mon.noname-a con 0x7f1c180af1d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:41.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180aed10 con 0x7f1c180af1d0 2024-03-29T10:56:41.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180afd50 con 0x7f1c1806d9d0 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180ab490 con 0x7f1c1813e820 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.726+0000 7f1c1e123640 10 monclient(hunting): authenticate will time out at 1521.019241s 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.728+0000 7f1c16ffd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f1c1813e820 0x7f1c1813ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:41408/0 (socket says 172.21.15.79:41408) 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.728+0000 7f1c16ffd640 1 -- 172.21.15.79:0/1324339810 learned_addr learned my addr 172.21.15.79:0/1324339810 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347495430 0 0) 0x7f1c180ab490 con 0x7f1c1813e820 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:41.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c00003520 con 0x7f1c1813e820 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745610422 0 0) 0x7f1c180aed10 con 0x7f1c180af1d0 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c180ab490 con 0x7f1c180af1d0 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3961795103 0 0) 0x7f1c00003520 con 0x7f1c1813e820 2024-03-29T10:56:41.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 >> v1:172.21.15.82:6789/0 conn(0x7f1c1806d9d0 legacy=0x7f1c180b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 >> v1:172.21.15.79:6789/0 conn(0x7f1c180af1d0 legacy=0x7f1c180af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 monclient: found mon.noname-b 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c180ab770 con 0x7f1c1813e820 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1c08001690 con 0x7f1c1813e820 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:41.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: dump: 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c1e123640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c1e123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 1 -- 172.21.15.79:0/1324339810 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c08001870 con 0x7f1c1813e820 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.729+0000 7f1c167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.730+0000 7f1c1e123640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:41.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.730+0000 7f1c1e123640 4 set_mon_vals no callback set 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.730+0000 7f1c1e123640 10 monclient: shutdown 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.730+0000 7f1c1e123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.730+0000 7f1c1e123640 1 -- 172.21.15.79:0/1324339810 >> v1:172.21.15.79:6790/0 conn(0x7f1c1813e820 legacy=0x7f1c1813ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.731+0000 7f1c1e123640 1 -- 172.21.15.79:0/1324339810 shutdown_connections 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.731+0000 7f1c1e123640 1 -- 172.21.15.79:0/1324339810 >> 172.21.15.79:0/1324339810 conn(0x7f1c1806b660 msgr2=0x7f1c1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.731+0000 7f1c1e123640 1 -- 172.21.15.79:0/1324339810 shutdown_connections 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.731+0000 7f1c1e123640 1 -- 172.21.15.79:0/1324339810 wait complete. 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.732+0000 7f1c1e123640 10 monclient: build_initial_monmap 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.732+0000 7f1c1e123640 10 monclient: monmap: 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:41.720 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.732+0000 7f1c1e123640 1 librados: starting msgr at 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.732+0000 7f1c1e123640 1 librados: starting objecter 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.733+0000 7f1c1e123640 1 Processor -- start 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.733+0000 7f1c1e123640 1 -- start start 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.733+0000 7f1c1e123640 1 librados: setting wanted keys 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.733+0000 7f1c1e123640 1 librados: calling monclient init 2024-03-29T10:56:41.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.733+0000 7f1c1e123640 10 monclient: init 2024-03-29T10:56:41.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:41.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:41.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient(hunting): picked mon.noname-b con 0x7f1c1806d9d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:41.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient(hunting): picked mon.noname-c con 0x7f1c180af1d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:41.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient(hunting): picked mon.noname-a con 0x7f1c1813e820 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:41.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180b4260 con 0x7f1c1813e820 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c181db5e0 con 0x7f1c1806d9d0 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c18140d10 con 0x7f1c180af1d0 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.734+0000 7f1c1e123640 10 monclient(hunting): authenticate will time out at 1521.027352s 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1c17fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f1c1813e820 0x7f1c181e6060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:34021/0 (socket says 172.21.15.79:34021) 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1c17fff640 1 -- 172.21.15.79:0/4086924669 learned_addr learned my addr 172.21.15.79:0/4086924669 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1bfffff640 10 client.?.objecter ms_handle_connect 0x7f1c1813e820 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1bfffff640 10 client.?.objecter ms_handle_connect 0x7f1c180af1d0 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1bfffff640 10 client.?.objecter ms_handle_connect 0x7f1c1806d9d0 2024-03-29T10:56:41.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.737+0000 7f1bfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:41.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.738+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038129656 0 0) 0x7f1c180b4260 con 0x7f1c1813e820 2024-03-29T10:56:41.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.738+0000 7f1bfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.738+0000 7f1bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.738+0000 7f1bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.738+0000 7f1bfffff640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.738+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c04004eb0 con 0x7f1c1813e820 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3829093068 0 0) 0x7f1c04004eb0 con 0x7f1c1813e820 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c180b4260 con 0x7f1c1813e820 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285418271 0 0) 0x7f1c181db5e0 con 0x7f1c1806d9d0 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c04004eb0 con 0x7f1c1806d9d0 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746742367 0 0) 0x7f1c18140d10 con 0x7f1c180af1d0 2024-03-29T10:56:41.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c181db5e0 con 0x7f1c180af1d0 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1c10001d10 con 0x7f1c1813e820 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): dump: 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:41.726 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2895037206 0 0) 0x7f1c180b4260 con 0x7f1c1813e820 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 >> v1:172.21.15.79:6790/0 conn(0x7f1c180af1d0 legacy=0x7f1c181dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 >> v1:172.21.15.82:6789/0 conn(0x7f1c1806d9d0 legacy=0x7f1c180b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 monclient: found mon.a 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c18140ea0 con 0x7f1c1813e820 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient: _finish_auth 0 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:41.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1bfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 5 monclient: authenticate success, global_id 4539 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 10 monclient: _renew_subs 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c181dc210 con 0x7f1c1813e820 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 10 monclient: _renew_subs 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c180b20c0 con 0x7f1c1813e820 2024-03-29T10:56:41.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.742+0000 7f1c1e123640 1 librados: init done 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.743+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c10001e70 con 0x7f1c1813e820 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.743+0000 7f1bfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.743+0000 7f1c15ffb640 4 set_mon_vals no callback set 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.744+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1c10003de0 con 0x7f1c1813e820 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.744+0000 7f1bfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.744+0000 7f1bfffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.744+0000 7f1bfffff640 10 monclient: dump: 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:41.729 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.745+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f1c10003f40 con 0x7f1c1813e820 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.745+0000 7f1c1e123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.745+0000 7f1c1e123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.745+0000 7f1c1e123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.745+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf8005080 con 0x7f1c1813e820 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.746+0000 7f1c177fe640 10 monclient: get_auth_request con 0x7f1c0406e050 auth_method 0 2024-03-29T10:56:41.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.747+0000 7f1bfffff640 10 client.4539.objecter ms_handle_connect 0x7f1c0406e050 2024-03-29T10:56:41.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.750+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f1c10054a60 con 0x7f1c1813e820 2024-03-29T10:56:41.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.750+0000 7f1bfffff640 10 client.4539.objecter ms_dispatch 0x7f1c18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:41.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.750+0000 7f1bfffff640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:41.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.750+0000 7f1bfffff640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:41.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.750+0000 7f1bfffff640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T10:56:41.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.755+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c10050570 con 0x7f1c1813e820 2024-03-29T10:56:41.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.755+0000 7f1bfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:41.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:41.755+0000 7f1bfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:42.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.075+0000 7f1c1e123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:42.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.075+0000 7f1c1e123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:42.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.075+0000 7f1c1e123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:56:42.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.075+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf8005330 con 0x7f1c1813e820 2024-03-29T10:56:42.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.076+0000 7f1bfffff640 1 -- 172.21.15.79:0/4086924669 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f1c10003c40 con 0x7f1c1813e820 2024-03-29T10:56:42.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.076+0000 7f1bfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:42.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.076+0000 7f1bfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:42.059 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:42.059 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:42.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 10 librados: watch_flush enter 2024-03-29T10:56:42.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 10 librados: watch_flush exit 2024-03-29T10:56:42.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T10:56:42.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 10 client.4539.objecter successfully canceled tick 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 >> v1:172.21.15.79:6816/4290694627 conn(0x7f1c0406e050 legacy=0x7f1c04070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 10 monclient: shutdown 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.080+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 >> v1:172.21.15.79:6789/0 conn(0x7f1c1813e820 legacy=0x7f1c181e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.081+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 shutdown_connections 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.081+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 >> 172.21.15.79:0/4086924669 conn(0x7f1c1806b660 msgr2=0x7f1c1813ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:42.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.081+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 shutdown_connections 2024-03-29T10:56:42.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.082+0000 7f1c1e123640 1 -- 172.21.15.79:0/4086924669 wait complete. 2024-03-29T10:56:42.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:42.082+0000 7f1c1e123640 1 librados: shutdown 2024-03-29T10:56:42.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:48.888 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.052+0000 7f77af040640 10 monclient: get_monmap_and_config 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.052+0000 7f77af040640 10 monclient: build_initial_monmap 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.052+0000 7f77af040640 10 monclient: monmap: 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:49.053254+0000 2024-03-29T10:56:49.033 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:49.053254+0000 2024-03-29T10:56:49.034 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:49.034 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.034 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:49.034 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:49.034 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:49.034 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.054+0000 7f77af040640 1 Processor -- start 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.054+0000 7f77af040640 1 -- start start 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.054+0000 7f77af040640 10 monclient: init 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient(hunting): picked mon.noname-c con 0x7f77a80b0860 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient(hunting): picked mon.noname-a con 0x7f77a813ef30 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:49.037 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient(hunting): picked mon.noname-b con 0x7f77a80af8e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80b0c40 con 0x7f77a813ef30 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80aed10 con 0x7f77a80b0860 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80afd50 con 0x7f77a80af8e0 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.055+0000 7f77af040640 10 monclient(hunting): authenticate will time out at 1528.347411s 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.056+0000 7f77ad5dd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f77a80af8e0 0x7f77a8149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:35616/0 (socket says 172.21.15.79:35616) 2024-03-29T10:56:49.038 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.056+0000 7f77ad5dd640 1 -- 172.21.15.79:0/2183504414 learned_addr learned my addr 172.21.15.79:0/2183504414 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:49.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.057+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115829494 0 0) 0x7f77a80afd50 con 0x7f77a80af8e0 2024-03-29T10:56:49.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.057+0000 7f77a77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:49.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.057+0000 7f77a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:49.039 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.057+0000 7f77a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:49.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.057+0000 7f77a77fe640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:56:49.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.057+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7790003520 con 0x7f77a80af8e0 2024-03-29T10:56:49.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.058+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441549614 0 0) 0x7f7790003520 con 0x7f77a80af8e0 2024-03-29T10:56:49.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.058+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 >> v1:172.21.15.82:6789/0 conn(0x7f77a80b0860 legacy=0x7f77a813e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:49.040 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.059+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 >> v1:172.21.15.79:6789/0 conn(0x7f77a813ef30 legacy=0x7f77a80af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80ab1b0 con 0x7f77a80af8e0 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:49.042 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f779c001690 con 0x7f77a80af8e0 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: dump: 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:49.043 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77af040640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77af040640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c001860 con 0x7f77a80af8e0 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 1 -- 172.21.15.79:0/2183504414 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f779c001cb0 con 0x7f77a80af8e0 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77a77fe640 10 monclient: dump: 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:49.044 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77af040640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.060+0000 7f77af040640 4 set_mon_vals no callback set 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.061+0000 7f77af040640 10 monclient: shutdown 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.061+0000 7f77af040640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.061+0000 7f77af040640 1 -- 172.21.15.79:0/2183504414 >> v1:172.21.15.79:6790/0 conn(0x7f77a80af8e0 legacy=0x7f77a8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.061+0000 7f77af040640 1 -- 172.21.15.79:0/2183504414 shutdown_connections 2024-03-29T10:56:49.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.061+0000 7f77af040640 1 -- 172.21.15.79:0/2183504414 >> 172.21.15.79:0/2183504414 conn(0x7f77a806b660 msgr2=0x7f77a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:49.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.065+0000 7f77af040640 1 -- 172.21.15.79:0/2183504414 shutdown_connections 2024-03-29T10:56:49.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.066+0000 7f77af040640 1 -- 172.21.15.79:0/2183504414 wait complete. 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.067+0000 7f77af040640 10 monclient: build_initial_monmap 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.067+0000 7f77af040640 10 monclient: monmap: 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.067+0000 7f77af040640 1 librados: starting msgr at 2024-03-29T10:56:49.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.068+0000 7f77af040640 1 librados: starting objecter 2024-03-29T10:56:49.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.068+0000 7f77af040640 1 Processor -- start 2024-03-29T10:56:49.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.069+0000 7f77af040640 1 -- start start 2024-03-29T10:56:49.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.069+0000 7f77af040640 1 librados: setting wanted keys 2024-03-29T10:56:49.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.069+0000 7f77af040640 1 librados: calling monclient init 2024-03-29T10:56:49.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.069+0000 7f77af040640 10 monclient: init 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient(hunting): picked mon.noname-b con 0x7f77a80af8e0 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient(hunting): picked mon.noname-a con 0x7f77a80b0860 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient(hunting): picked mon.noname-c con 0x7f77a813ef30 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80b3840 con 0x7f77a80b0860 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80b3b20 con 0x7f77a80af8e0 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a81c77d0 con 0x7f77a813ef30 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77af040640 10 monclient(hunting): authenticate will time out at 1528.362735s 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77ad5dd640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f77a813ef30 0x7f77a81c8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:38860/0 (socket says 172.21.15.79:38860) 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.070+0000 7f77ad5dd640 1 -- 172.21.15.79:0/2073416868 learned_addr learned my addr 172.21.15.79:0/2073416868 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 10 client.?.objecter ms_handle_connect 0x7f77a813ef30 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:49.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543203457 0 0) 0x7f77a81c77d0 con 0x7f77a813ef30 2024-03-29T10:56:49.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:49.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:49.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:49.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:56:49.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.071+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7794004eb0 con 0x7f77a813ef30 2024-03-29T10:56:49.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.074+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2285737101 0 0) 0x7f7794004eb0 con 0x7f77a813ef30 2024-03-29T10:56:49.055 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.074+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77a81c77d0 con 0x7f77a813ef30 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.075+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f779c001b70 con 0x7f77a813ef30 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.075+0000 7f77a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.075+0000 7f77a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.075+0000 7f77a57fa640 10 monclient(hunting): dump: 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.056 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.075+0000 7f77a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:49.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.076+0000 7f77a57fa640 10 client.?.objecter ms_handle_connect 0x7f77a80af8e0 2024-03-29T10:56:49.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.076+0000 7f77a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:49.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.076+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294481483 0 0) 0x7f77a80b3b20 con 0x7f77a80af8e0 2024-03-29T10:56:49.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7794004eb0 con 0x7f77a80af8e0 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 client.?.objecter ms_handle_connect 0x7f77a80b0860 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613439939 0 0) 0x7f77a80b3840 con 0x7f77a80b0860 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a80b3b20 con 0x7f77a80b0860 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.077+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327846571 0 0) 0x7f7794004eb0 con 0x7f77a80af8e0 2024-03-29T10:56:49.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77a80b3840 con 0x7f77a80af8e0 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f7798001790 con 0x7f77a80af8e0 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 10 monclient(hunting): dump: 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2737369749 0 0) 0x7f77a81c77d0 con 0x7f77a813ef30 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 >> v1:172.21.15.82:6789/0 conn(0x7f77a80af8e0 legacy=0x7f77a8132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 >> v1:172.21.15.79:6789/0 conn(0x7f77a80b0860 legacy=0x7f77a81c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 monclient: found mon.c 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.078+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80b4020 con 0x7f77a813ef30 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77a57fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77a57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 10 monclient: _renew_subs 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:49.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a80b14c0 con 0x7f77a813ef30 2024-03-29T10:56:49.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:49.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 10 monclient: _renew_subs 2024-03-29T10:56:49.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:49.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a8133070 con 0x7f77a813ef30 2024-03-29T10:56:49.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.079+0000 7f77af040640 1 librados: init done 2024-03-29T10:56:49.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c004bf0 con 0x7f77a813ef30 2024-03-29T10:56:49.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:49.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f779c004f00 con 0x7f77a813ef30 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a57fa640 10 monclient: dump: 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:49.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77af040640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77af040640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77af040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77af040640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a81c7960 con 0x7f77a813ef30 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.080+0000 7f77a67fc640 4 set_mon_vals no callback set 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.082+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f779c01aed0 con 0x7f77a813ef30 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.083+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f779c084bf0 con 0x7f77a813ef30 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.083+0000 7f77a57fa640 10 client.4625.objecter ms_dispatch 0x7f77a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.083+0000 7f77a57fa640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.083+0000 7f77a57fa640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.083+0000 7f77a57fa640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.083+0000 7f77acddc640 10 monclient: get_auth_request con 0x7f779406dda0 auth_method 0 2024-03-29T10:56:49.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.084+0000 7f77a57fa640 10 client.4625.objecter ms_handle_connect 0x7f779406dda0 2024-03-29T10:56:49.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.092+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f779c0504d0 con 0x7f77a813ef30 2024-03-29T10:56:49.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.092+0000 7f77a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:49.073 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.092+0000 7f77a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:49.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.393+0000 7f77af040640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:49.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.393+0000 7f77af040640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:49.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.393+0000 7f77af040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:49.374 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.393+0000 7f77af040640 1 -- 172.21.15.79:0/2073416868 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77a8075f10 con 0x7f77a813ef30 2024-03-29T10:56:49.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.394+0000 7f77a57fa640 1 -- 172.21.15.79:0/2073416868 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f779c052400 con 0x7f77a813ef30 2024-03-29T10:56:49.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.394+0000 7f77a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:49.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.394+0000 7f77a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:49.375 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:49.375 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:49.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 10 librados: watch_flush enter 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 10 librados: watch_flush exit 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 10 client.4625.objecter successfully canceled tick 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 1 -- 172.21.15.79:0/2073416868 >> v1:172.21.15.79:6816/4290694627 conn(0x7f779406dda0 legacy=0x7f7794070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 10 monclient: shutdown 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 1 -- 172.21.15.79:0/2073416868 >> v1:172.21.15.79:6790/0 conn(0x7f77a813ef30 legacy=0x7f77a81c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.397+0000 7f778effd640 1 -- 172.21.15.79:0/2073416868 shutdown_connections 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.398+0000 7f778effd640 1 -- 172.21.15.79:0/2073416868 >> 172.21.15.79:0/2073416868 conn(0x7f77a806b660 msgr2=0x7f77a806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.398+0000 7f778effd640 1 -- 172.21.15.79:0/2073416868 shutdown_connections 2024-03-29T10:56:49.379 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.398+0000 7f778effd640 1 -- 172.21.15.79:0/2073416868 wait complete. 2024-03-29T10:56:49.380 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:49.398+0000 7f778effd640 1 librados: shutdown 2024-03-29T10:56:49.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:56.209 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:56.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.379+0000 7fc88e28b640 10 monclient: get_monmap_and_config 2024-03-29T10:56:56.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.379+0000 7fc88e28b640 10 monclient: build_initial_monmap 2024-03-29T10:56:56.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.379+0000 7fc88e28b640 10 monclient: monmap: 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:56:56.380407+0000 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:56:56.380407+0000 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:56:56.361 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:56.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.380+0000 7fc88e28b640 1 Processor -- start 2024-03-29T10:56:56.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.380+0000 7fc88e28b640 1 -- start start 2024-03-29T10:56:56.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.380+0000 7fc88e28b640 10 monclient: init 2024-03-29T10:56:56.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:56.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:56.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient(hunting): picked mon.noname-c con 0x7fc8880b4100 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:56.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient(hunting): picked mon.noname-b con 0x7fc88813ee90 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:56.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8880af900 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:56.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8880afd10 con 0x7fc8880af900 2024-03-29T10:56:56.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8880af6f0 con 0x7fc8880b4100 2024-03-29T10:56:56.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8880aaf60 con 0x7fc88813ee90 2024-03-29T10:56:56.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:56.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.381+0000 7fc88e28b640 10 monclient(hunting): authenticate will time out at 1535.673721s 2024-03-29T10:56:56.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc8877fe640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fc88813ee90 0x7fc8880adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:48996/0 (socket says 172.21.15.79:48996) 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc8877fe640 1 -- 172.21.15.79:0/3300141171 learned_addr learned my addr 172.21.15.79:0/3300141171 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431341286 0 0) 0x7fc8880aaf60 con 0x7fc88813ee90 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc886ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.383+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc870003520 con 0x7fc88813ee90 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.384+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286218151 0 0) 0x7fc870003520 con 0x7fc88813ee90 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.384+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 >> v1:172.21.15.82:6789/0 conn(0x7fc8880b4100 legacy=0x7fc8880a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:56.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.384+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 >> v1:172.21.15.79:6789/0 conn(0x7fc8880af900 legacy=0x7fc88813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 1 monclient: found mon.noname-b 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88813e970 con 0x7fc88813ee90 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc88e28b640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc88e28b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:56.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc87c001690 con 0x7fc88813ee90 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: dump: 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:56.369 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc88e28b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:56.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc87c001920 con 0x7fc88813ee90 2024-03-29T10:56:56.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:56.370 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc88e28b640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.387+0000 7fc88e28b640 4 set_mon_vals no callback set 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc886ffd640 1 -- 172.21.15.79:0/3300141171 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc87c001cb0 con 0x7fc88813ee90 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc886ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc886ffd640 10 monclient: dump: 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:56.371 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc88e28b640 10 monclient: shutdown 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc88e28b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc88e28b640 1 -- 172.21.15.79:0/3300141171 >> v1:172.21.15.79:6790/0 conn(0x7fc88813ee90 legacy=0x7fc8880adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc88e28b640 1 -- 172.21.15.79:0/3300141171 shutdown_connections 2024-03-29T10:56:56.372 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.388+0000 7fc88e28b640 1 -- 172.21.15.79:0/3300141171 >> 172.21.15.79:0/3300141171 conn(0x7fc8880b35e0 msgr2=0x7fc8880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:56.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.390+0000 7fc88e28b640 1 -- 172.21.15.79:0/3300141171 shutdown_connections 2024-03-29T10:56:56.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.390+0000 7fc88e28b640 1 -- 172.21.15.79:0/3300141171 wait complete. 2024-03-29T10:56:56.375 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.391+0000 7fc88e28b640 10 monclient: build_initial_monmap 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.391+0000 7fc88e28b640 10 monclient: monmap: 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 1 librados: starting msgr at 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 1 librados: starting objecter 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 1 Processor -- start 2024-03-29T10:56:56.376 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 1 -- start start 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 1 librados: setting wanted keys 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 1 librados: calling monclient init 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.392+0000 7fc88e28b640 10 monclient: init 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient(hunting): picked mon.noname-a con 0x7fc8880af900 addr v1:172.21.15.79:6789/0 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient(hunting): picked mon.noname-b con 0x7fc8880b4100 addr v1:172.21.15.82:6789/0 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient(hunting): picked mon.noname-c con 0x7fc88813ee90 addr v1:172.21.15.79:6790/0 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc888071bc0 con 0x7fc8880af900 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88812a0d0 con 0x7fc8880b4100 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88812a3b0 con 0x7fc88813ee90 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88e28b640 10 monclient(hunting): authenticate will time out at 1535.685380s 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88c828640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fc88813ee90 0x7fc8881e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:50492/0 (socket says 172.21.15.79:50492) 2024-03-29T10:56:56.377 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc88c828640 1 -- 172.21.15.79:0/2628680484 learned_addr learned my addr 172.21.15.79:0/2628680484 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 10 client.?.objecter ms_handle_connect 0x7fc88813ee90 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235282707 0 0) 0x7fc88812a3b0 con 0x7fc88813ee90 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:56:56.378 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.393+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc874004eb0 con 0x7fc88813ee90 2024-03-29T10:56:56.382 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1651145691 0 0) 0x7fc874004eb0 con 0x7fc88813ee90 2024-03-29T10:56:56.382 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc88812a3b0 con 0x7fc88813ee90 2024-03-29T10:56:56.382 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc8780018e0 con 0x7fc88813ee90 2024-03-29T10:56:56.382 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:56.382 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.396+0000 7fc884ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 client.?.objecter ms_handle_connect 0x7fc8880b4100 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:56.383 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238863970 0 0) 0x7fc88812a0d0 con 0x7fc8880b4100 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc874004eb0 con 0x7fc8880b4100 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2089828666 0 0) 0x7fc88812a3b0 con 0x7fc88813ee90 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 >> v1:172.21.15.82:6789/0 conn(0x7fc8880b4100 legacy=0x7fc888072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 >> v1:172.21.15.79:6789/0 conn(0x7fc8880af900 legacy=0x7fc88806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 monclient: found mon.c 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc888128fd0 con 0x7fc88813ee90 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:56.384 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 10 monclient: _renew_subs 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc888071750 con 0x7fc88813ee90 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 10 monclient: _renew_subs 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc888141bf0 con 0x7fc88813ee90 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc88e28b640 1 librados: init done 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8780035f0 con 0x7fc88813ee90 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:56.385 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fc878003a00 con 0x7fc88813ee90 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc884ff9640 10 monclient: dump: 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.397+0000 7fc8867fc640 4 set_mon_vals no callback set 2024-03-29T10:56:56.386 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.400+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fc878003b60 con 0x7fc88813ee90 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fc878084a30 con 0x7fc88813ee90 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc884ff9640 10 client.4631.objecter ms_dispatch 0x7fc888000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc884ff9640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc884ff9640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc884ff9640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc88e28b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.401+0000 7fc887fff640 10 monclient: get_auth_request con 0x7fc87406dcb0 auth_method 0 2024-03-29T10:56:56.387 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.402+0000 7fc884ff9640 10 client.4631.objecter ms_handle_connect 0x7fc87406dcb0 2024-03-29T10:56:56.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.404+0000 7fc88e28b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:56.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.405+0000 7fc88e28b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:56.388 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.405+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc868005080 con 0x7fc88813ee90 2024-03-29T10:56:56.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.414+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc878050370 con 0x7fc88813ee90 2024-03-29T10:56:56.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.414+0000 7fc884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:56.394 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.414+0000 7fc884ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:56.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.748+0000 7fc88e28b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:56.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.748+0000 7fc88e28b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:56.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.748+0000 7fc88e28b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:56:56.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.748+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc868005330 con 0x7fc88813ee90 2024-03-29T10:56:56.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.749+0000 7fc884ff9640 1 -- 172.21.15.79:0/2628680484 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fc8780840e0 con 0x7fc88813ee90 2024-03-29T10:56:56.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.749+0000 7fc884ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:56.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.749+0000 7fc884ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:56.731 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:56:56.732 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:56.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 10 librados: watch_flush enter 2024-03-29T10:56:56.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 10 librados: watch_flush exit 2024-03-29T10:56:56.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T10:56:56.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 10 client.4631.objecter successfully canceled tick 2024-03-29T10:56:56.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 >> v1:172.21.15.79:6816/4290694627 conn(0x7fc87406dcb0 legacy=0x7fc874070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:56.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 10 monclient: shutdown 2024-03-29T10:56:56.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:56.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.753+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 >> v1:172.21.15.79:6790/0 conn(0x7fc88813ee90 legacy=0x7fc8881e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:56.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.754+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 shutdown_connections 2024-03-29T10:56:56.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.754+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 >> 172.21.15.79:0/2628680484 conn(0x7fc8880b35e0 msgr2=0x7fc8880b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:56.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.754+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 shutdown_connections 2024-03-29T10:56:56.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.754+0000 7fc88e28b640 1 -- 172.21.15.79:0/2628680484 wait complete. 2024-03-29T10:56:56.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:56:56.754+0000 7fc88e28b640 1 librados: shutdown 2024-03-29T10:56:56.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:03.558 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:03.706 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.726+0000 7fdb7fa8d640 10 monclient: get_monmap_and_config 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.726+0000 7fdb7fa8d640 10 monclient: build_initial_monmap 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.726+0000 7fdb7fa8d640 10 monclient: monmap: 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:03.727025+0000 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:03.727025+0000 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:03.707 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:03.708 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:03.708 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.728+0000 7fdb7fa8d640 1 Processor -- start 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.728+0000 7fdb7fa8d640 1 -- start start 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.728+0000 7fdb7fa8d640 10 monclient: init 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb780708a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb7806fe90 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb78143c50 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:03.712 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb78072630 con 0x7fdb78143c50 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb78071410 con 0x7fdb7806fe90 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780afaa0 con 0x7fdb780708a0 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.729+0000 7fdb7fa8d640 10 monclient(hunting): authenticate will time out at 1543.021819s 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7d829640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fdb780708a0 0x7fdb780af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:47834/0 (socket says 172.21.15.79:47834) 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7d829640 1 -- 172.21.15.79:0/3832923532 learned_addr learned my addr 172.21.15.79:0/3832923532 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399006991 0 0) 0x7fdb780afaa0 con 0x7fdb780708a0 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7c827640 10 monclient(hunting): _negotiate 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7c827640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7c827640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7c827640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.730+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb64003520 con 0x7fdb780708a0 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.731+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1449272454 0 0) 0x7fdb64003520 con 0x7fdb780708a0 2024-03-29T10:57:03.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.731+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 >> v1:172.21.15.82:6789/0 conn(0x7fdb7806fe90 legacy=0x7fdb78143540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:03.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 >> v1:172.21.15.79:6789/0 conn(0x7fdb78143c50 legacy=0x7fdb7814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: _finish_hunting 0 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 1 monclient: found mon.noname-b 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780ac8b0 con 0x7fdb780708a0 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: _finish_auth 0 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: _check_auth_tickets 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdb68001690 con 0x7fdb780708a0 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: dump: 2024-03-29T10:57:03.716 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7fa8d640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7fa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb68001860 con 0x7fdb780708a0 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 1 -- 172.21.15.79:0/3832923532 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdb68001c70 con 0x7fdb780708a0 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:03.717 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7c827640 10 monclient: dump: 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7fa8d640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.734+0000 7fdb7fa8d640 4 set_mon_vals no callback set 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.735+0000 7fdb7fa8d640 10 monclient: shutdown 2024-03-29T10:57:03.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.735+0000 7fdb7fa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:03.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.735+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/3832923532 >> v1:172.21.15.79:6790/0 conn(0x7fdb780708a0 legacy=0x7fdb780af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:03.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.737+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/3832923532 shutdown_connections 2024-03-29T10:57:03.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.737+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/3832923532 >> 172.21.15.79:0/3832923532 conn(0x7fdb780675b0 msgr2=0x7fdb780b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:03.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.738+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/3832923532 shutdown_connections 2024-03-29T10:57:03.720 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.739+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/3832923532 wait complete. 2024-03-29T10:57:03.721 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.740+0000 7fdb7fa8d640 10 monclient: build_initial_monmap 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.740+0000 7fdb7fa8d640 10 monclient: monmap: 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.740+0000 7fdb7fa8d640 1 librados: starting msgr at 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.740+0000 7fdb7fa8d640 1 librados: starting objecter 2024-03-29T10:57:03.722 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.741+0000 7fdb7fa8d640 1 Processor -- start 2024-03-29T10:57:03.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.741+0000 7fdb7fa8d640 1 -- start start 2024-03-29T10:57:03.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.741+0000 7fdb7fa8d640 1 librados: setting wanted keys 2024-03-29T10:57:03.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.741+0000 7fdb7fa8d640 1 librados: calling monclient init 2024-03-29T10:57:03.723 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.741+0000 7fdb7fa8d640 10 monclient: init 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb7806fe90 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb780708a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb78143c50 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780abf60 con 0x7fdb780708a0 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780ac240 con 0x7fdb7806fe90 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7813d340 con 0x7fdb78143c50 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:03.724 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.742+0000 7fdb7fa8d640 10 monclient(hunting): authenticate will time out at 1543.034321s 2024-03-29T10:57:03.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.744+0000 7fdb7e02a640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fdb78143c50 0x7fdb780a9fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:49606/0 (socket says 172.21.15.79:49606) 2024-03-29T10:57:03.725 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.744+0000 7fdb7e02a640 1 -- 172.21.15.79:0/2697453342 learned_addr learned my addr 172.21.15.79:0/2697453342 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:03.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.745+0000 7fdb6e7fc640 10 client.?.objecter ms_handle_connect 0x7fdb78143c50 2024-03-29T10:57:03.726 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.745+0000 7fdb6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843784828 0 0) 0x7fdb7813d340 con 0x7fdb78143c50 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c004eb0 con 0x7fdb78143c50 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 client.?.objecter ms_handle_connect 0x7fdb780708a0 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:03.727 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1911911227 0 0) 0x7fdb5c004eb0 con 0x7fdb78143c50 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb7813d340 con 0x7fdb78143c50 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700646839 0 0) 0x7fdb780abf60 con 0x7fdb780708a0 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c004eb0 con 0x7fdb780708a0 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdb70001c60 con 0x7fdb78143c50 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.746+0000 7fdb6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.728 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1078064639 0 0) 0x7fdb7813d340 con 0x7fdb78143c50 2024-03-29T10:57:03.729 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 >> v1:172.21.15.82:6789/0 conn(0x7fdb7806fe90 legacy=0x7fdb7813cc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:03.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 >> v1:172.21.15.79:6789/0 conn(0x7fdb780708a0 legacy=0x7fdb781db9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:03.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 1 monclient: found mon.c 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7813d4d0 con 0x7fdb78143c50 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.747+0000 7fdb6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 5 monclient: authenticate success, global_id 4637 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 10 monclient: _renew_subs 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb781dc0d0 con 0x7fdb78143c50 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 10 monclient: _renew_subs 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb780aa8b0 con 0x7fdb78143c50 2024-03-29T10:57:03.731 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb7fa8d640 1 librados: init done 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb70001e30 con 0x7fdb78143c50 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fdb70003c40 con 0x7fdb78143c50 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.750+0000 7fdb6e7fc640 10 monclient: dump: 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:03.732 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:03.733 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:03.733 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.751+0000 7fdb6ffff640 4 set_mon_vals no callback set 2024-03-29T10:57:03.733 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.752+0000 7fdb7fa8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:03.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.752+0000 7fdb7fa8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:03.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.752+0000 7fdb7fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:03.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.752+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60005080 con 0x7fdb78143c50 2024-03-29T10:57:03.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.753+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fdb70003da0 con 0x7fdb78143c50 2024-03-29T10:57:03.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.754+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fdb70084de0 con 0x7fdb78143c50 2024-03-29T10:57:03.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.754+0000 7fdb6e7fc640 10 client.4637.objecter ms_dispatch 0x7fdb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:03.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.754+0000 7fdb6e7fc640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:03.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.754+0000 7fdb6e7fc640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:03.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.754+0000 7fdb6e7fc640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T10:57:03.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.755+0000 7fdb7d829640 10 monclient: get_auth_request con 0x7fdb5c06db20 auth_method 0 2024-03-29T10:57:03.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.755+0000 7fdb6e7fc640 10 client.4637.objecter ms_handle_connect 0x7fdb5c06db20 2024-03-29T10:57:03.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.761+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb700b48b0 con 0x7fdb78143c50 2024-03-29T10:57:03.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.761+0000 7fdb6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:03.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:03.761+0000 7fdb6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb7fa8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb7fa8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb7fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb60005330 con 0x7fdb78143c50 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb6e7fc640 1 -- 172.21.15.79:0/2697453342 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fdb700504f0 con 0x7fdb78143c50 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:04.069 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.088+0000 7fdb6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:04.070 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:04.070 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:04.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 10 librados: watch_flush enter 2024-03-29T10:57:04.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 10 librados: watch_flush exit 2024-03-29T10:57:04.075 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 10 client.4637.objecter successfully canceled tick 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 >> v1:172.21.15.79:6816/4290694627 conn(0x7fdb5c06db20 legacy=0x7fdb5c06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 10 monclient: shutdown 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.093+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 >> v1:172.21.15.79:6790/0 conn(0x7fdb78143c50 legacy=0x7fdb780a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.094+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 shutdown_connections 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.094+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 >> 172.21.15.79:0/2697453342 conn(0x7fdb780675b0 msgr2=0x7fdb780b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.095+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 shutdown_connections 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.095+0000 7fdb7fa8d640 1 -- 172.21.15.79:0/2697453342 wait complete. 2024-03-29T10:57:04.076 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:04.095+0000 7fdb7fa8d640 1 librados: shutdown 2024-03-29T10:57:04.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:10.899 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:11.045 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.065+0000 7f201af64640 10 monclient: get_monmap_and_config 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.065+0000 7f201af64640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.065+0000 7f201af64640 10 monclient: monmap: 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:11.066130+0000 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:11.066130+0000 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:11.046 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.066+0000 7f201af64640 1 Processor -- start 2024-03-29T10:57:11.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.066+0000 7f201af64640 1 -- start start 2024-03-29T10:57:11.047 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.066+0000 7f201af64640 10 monclient: init 2024-03-29T10:57:11.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:11.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient(hunting): picked mon.noname-a con 0x7f20140b2800 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:11.049 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient(hunting): picked mon.noname-b con 0x7f2014142b20 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient(hunting): picked mon.noname-c con 0x7f201413c040 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140aad10 con 0x7f20140b2800 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2014141f30 con 0x7f201413c040 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140aaf60 con 0x7f2014142b20 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f201af64640 10 monclient(hunting): authenticate will time out at 1550.359837s 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f2019f62640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f20140b2800 0x7f2014142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:41460/0 (socket says 172.21.15.79:41460) 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.067+0000 7f2019f62640 1 -- 172.21.15.79:0/3111065072 learned_addr learned my addr 172.21.15.79:0/3111065072 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064558930 0 0) 0x7f20140aad10 con 0x7f20140b2800 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.050 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2000003520 con 0x7f20140b2800 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871574163 0 0) 0x7f2014141f30 con 0x7f201413c040 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20140aad10 con 0x7f201413c040 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214853356 0 0) 0x7f20140aaf60 con 0x7f2014142b20 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2014141f30 con 0x7f2014142b20 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1888185938 0 0) 0x7f2014141f30 con 0x7f2014142b20 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 >> v1:172.21.15.82:6789/0 conn(0x7f201413c040 legacy=0x7f201414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.051 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 >> v1:172.21.15.79:6789/0 conn(0x7f20140b2800 legacy=0x7f2014142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 monclient: found mon.noname-b 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20140b3900 con 0x7f2014142b20 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2004001530 con 0x7f2014142b20 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.068+0000 7f2018f60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 10 monclient: dump: 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:11.052 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f201af64640 5 monclient: authenticate success, global_id 4640 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f201af64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2004001690 con 0x7f2014142b20 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 1 -- 172.21.15.79:0/3111065072 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2004001ae0 con 0x7f2014142b20 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f2018f60640 10 monclient: dump: 2024-03-29T10:57:11.053 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f201af64640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:11.054 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f201af64640 4 set_mon_vals no callback set 2024-03-29T10:57:11.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.069+0000 7f201af64640 10 monclient: shutdown 2024-03-29T10:57:11.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.070+0000 7f201af64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.070+0000 7f201af64640 1 -- 172.21.15.79:0/3111065072 >> v1:172.21.15.79:6790/0 conn(0x7f2014142b20 legacy=0x7f201413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.070+0000 7f201af64640 1 -- 172.21.15.79:0/3111065072 shutdown_connections 2024-03-29T10:57:11.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.070+0000 7f201af64640 1 -- 172.21.15.79:0/3111065072 >> 172.21.15.79:0/3111065072 conn(0x7f201406d0e0 msgr2=0x7f2014066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.057 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.071+0000 7f201af64640 1 -- 172.21.15.79:0/3111065072 shutdown_connections 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.071+0000 7f201af64640 1 -- 172.21.15.79:0/3111065072 wait complete. 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.071+0000 7f201af64640 10 monclient: build_initial_monmap 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.071+0000 7f201af64640 10 monclient: monmap: 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.058 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 1 librados: starting msgr at 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 1 librados: starting objecter 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 1 Processor -- start 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 1 -- start start 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 1 librados: setting wanted keys 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 1 librados: calling monclient init 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.072+0000 7f201af64640 10 monclient: init 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.073+0000 7f201af64640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.073+0000 7f201af64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.073+0000 7f201af64640 10 monclient(hunting): picked mon.noname-a con 0x7f20140b2800 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 10 monclient(hunting): picked mon.noname-b con 0x7f201413c040 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 10 monclient(hunting): picked mon.noname-c con 0x7f2014145a80 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140af0b0 con 0x7f20140b2800 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140ad7e0 con 0x7f201413c040 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20140af300 con 0x7f2014145a80 2024-03-29T10:57:11.059 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f201af64640 10 monclient(hunting): authenticate will time out at 1550.366198s 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f2019f62640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f20140b2800 0x7f20141da8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:41666/0 (socket says 172.21.15.79:41666) 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f2019f62640 1 -- 172.21.15.79:0/2621282151 learned_addr learned my addr 172.21.15.79:0/2621282151 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 client.?.objecter ms_handle_connect 0x7f20140b2800 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 client.?.objecter ms_handle_connect 0x7f2014145a80 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 client.?.objecter ms_handle_connect 0x7f201413c040 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154273720 0 0) 0x7f20140af0b0 con 0x7f20140b2800 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff8004eb0 con 0x7f20140b2800 2024-03-29T10:57:11.060 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684438493 0 0) 0x7f20140af300 con 0x7f2014145a80 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.074+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20140af0b0 con 0x7f2014145a80 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355316479 0 0) 0x7f20140ad7e0 con 0x7f201413c040 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20140af300 con 0x7f201413c040 2024-03-29T10:57:11.061 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3662735370 0 0) 0x7f1ff8004eb0 con 0x7f20140b2800 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20140ad7e0 con 0x7f20140b2800 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f20100019a0 con 0x7f20140b2800 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:11.062 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602733393 0 0) 0x7f20140af0b0 con 0x7f2014145a80 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ff8004eb0 con 0x7f2014145a80 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f200c001b00 con 0x7f2014145a80 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:11.063 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 738315073 0 0) 0x7f20140ad7e0 con 0x7f20140b2800 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 >> v1:172.21.15.79:6790/0 conn(0x7f2014145a80 legacy=0x7f20140acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 >> v1:172.21.15.82:6789/0 conn(0x7f201413c040 legacy=0x7f20140adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 monclient: found mon.a 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201413a020 con 0x7f20140b2800 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 5 monclient: authenticate success, global_id 4551 2024-03-29T10:57:11.064 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 10 monclient: _renew_subs 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201413a2d0 con 0x7f20140b2800 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 10 monclient: _renew_subs 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f201413a800 con 0x7f20140b2800 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f201af64640 1 librados: init done 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20100047e0 con 0x7f20140b2800 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f2010004af0 con 0x7f20140b2800 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200bfff640 4 set_mon_vals no callback set 2024-03-29T10:57:11.065 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.075+0000 7f200a7fc640 10 monclient: dump: 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.077+0000 7f201af64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.077+0000 7f201af64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.077+0000 7f201af64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:11.066 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.077+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc005080 con 0x7f20140b2800 2024-03-29T10:57:11.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.078+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f201001aac0 con 0x7f20140b2800 2024-03-29T10:57:11.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.079+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f2010084a70 con 0x7f20140b2800 2024-03-29T10:57:11.067 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.079+0000 7f200a7fc640 10 client.4551.objecter ms_dispatch 0x7f2014000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.079+0000 7f200a7fc640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.079+0000 7f200a7fc640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.079+0000 7f200a7fc640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.080+0000 7f2019761640 10 monclient: get_auth_request con 0x7f1ff806df30 auth_method 0 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.080+0000 7f200a7fc640 10 client.4551.objecter ms_handle_connect 0x7f1ff806df30 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.086+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2010050350 con 0x7f20140b2800 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.086+0000 7f200a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:11.068 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.086+0000 7f200a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:11.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f201af64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f201af64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f201af64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ffc005330 con 0x7f20140b2800 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f200a7fc640 1 -- 172.21.15.79:0/2621282151 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f2010052280 con 0x7f20140b2800 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f200a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.454+0000 7f200a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:11.435 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 10 librados: watch_flush enter 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 10 librados: watch_flush exit 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 10 client.4551.objecter successfully canceled tick 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 >> v1:172.21.15.79:6816/4290694627 conn(0x7f1ff806df30 legacy=0x7f1ff80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 10 monclient: shutdown 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 >> v1:172.21.15.79:6789/0 conn(0x7f20140b2800 legacy=0x7f20141da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 shutdown_connections 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 >> 172.21.15.79:0/2621282151 conn(0x7f201406d0e0 msgr2=0x7f20140b82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 shutdown_connections 2024-03-29T10:57:11.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 -- 172.21.15.79:0/2621282151 wait complete. 2024-03-29T10:57:11.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:11.458+0000 7f201af64640 1 librados: shutdown 2024-03-29T10:57:11.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:18.262 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:18.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.436+0000 7fbcc0e0f640 10 monclient: get_monmap_and_config 2024-03-29T10:57:18.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.436+0000 7fbcc0e0f640 10 monclient: build_initial_monmap 2024-03-29T10:57:18.425 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.437+0000 7fbcc0e0f640 10 monclient: monmap: 2024-03-29T10:57:18.425 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:18.426 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:18.426 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:18.437889+0000 2024-03-29T10:57:18.426 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:18.437889+0000 2024-03-29T10:57:18.426 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:18.426 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.426 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:18.427 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:18.427 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:18.427 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.437+0000 7fbcc0e0f640 1 Processor -- start 2024-03-29T10:57:18.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.437+0000 7fbcc0e0f640 1 -- start start 2024-03-29T10:57:18.433 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.437+0000 7fbcc0e0f640 10 monclient: init 2024-03-29T10:57:18.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:18.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:18.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient(hunting): picked mon.noname-b con 0x7fbcbc1457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:18.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient(hunting): picked mon.noname-a con 0x7fbcbc13f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:18.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient(hunting): picked mon.noname-c con 0x7fbcbc0a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:18.434 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc0aaef0 con 0x7fbcbc13f650 2024-03-29T10:57:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc145bd0 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc145eb0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.438+0000 7fbcc0e0f640 10 monclient(hunting): authenticate will time out at 1557.731056s 2024-03-29T10:57:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcbad9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fbcbc0a9d70 0x7fbcbc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:33817/0 (socket says 172.21.15.79:33817) 2024-03-29T10:57:18.435 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcbad9d640 1 -- 172.21.15.79:0/1026662776 learned_addr learned my addr 172.21.15.79:0/1026662776 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:18.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594711372 0 0) 0x7fbcbc145bd0 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcb959a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:18.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcb959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:18.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:18.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcb959a640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:57:18.436 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.442+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca4003520 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.443+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975707309 0 0) 0x7fbca4003520 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.443+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 >> v1:172.21.15.79:6790/0 conn(0x7fbcbc1457f0 legacy=0x7fbcbc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.443+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 >> v1:172.21.15.79:6789/0 conn(0x7fbcbc13f650 legacy=0x7fbcbc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 1 monclient: found mon.noname-c 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:57:18.437 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc146340 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: _finish_auth 0 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbcb4001a10 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:18.438 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: dump: 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcc0e0f640 5 monclient: authenticate success, global_id 4498 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcc0e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:18.439 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcb4001be0 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 1 -- 172.21.15.79:0/1026662776 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbcb4002030 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcb959a640 10 monclient: dump: 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.440 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcc0e0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.446+0000 7fbcc0e0f640 4 set_mon_vals no callback set 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 10 monclient: shutdown 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/1026662776 >> v1:172.21.15.82:6789/0 conn(0x7fbcbc0a9d70 legacy=0x7fbcbc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/1026662776 shutdown_connections 2024-03-29T10:57:18.441 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/1026662776 >> 172.21.15.79:0/1026662776 conn(0x7fbcbc066a00 msgr2=0x7fbcbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/1026662776 shutdown_connections 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.448+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/1026662776 wait complete. 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.449+0000 7fbcc0e0f640 10 monclient: build_initial_monmap 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.449+0000 7fbcc0e0f640 10 monclient: monmap: 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:18.442 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.449+0000 7fbcc0e0f640 1 librados: starting msgr at 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.449+0000 7fbcc0e0f640 1 librados: starting objecter 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 Processor -- start 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 -- start start 2024-03-29T10:57:18.443 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 librados: setting wanted keys 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 librados: calling monclient init 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient: init 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient(hunting): picked mon.noname-a con 0x7fbcbc0a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient(hunting): picked mon.noname-c con 0x7fbcbc13f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient(hunting): picked mon.noname-b con 0x7fbcbc1457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc1428b0 con 0x7fbcbc0a9d70 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc06f3d0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc06f640 con 0x7fbcbc13f650 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:18.444 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.450+0000 7fbcc0e0f640 10 monclient(hunting): authenticate will time out at 1557.743061s 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcbad9d640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fbcbc1457f0 0x7fbcbc137810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:33891/0 (socket says 172.21.15.79:33891) 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcbad9d640 1 -- 172.21.15.79:0/667969197 learned_addr learned my addr 172.21.15.79:0/667969197 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 client.?.objecter ms_handle_connect 0x7fbcbc1457f0 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 client.?.objecter ms_handle_connect 0x7fbcbc13f650 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:18.448 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337230890 0 0) 0x7fbcbc06f3d0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:18.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:18.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:18.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:57:18.449 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c004eb0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875780380 0 0) 0x7fbcbc06f640 con 0x7fbcbc13f650 2024-03-29T10:57:18.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:18.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:18.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:18.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:57:18.450 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.458+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcbc06f3d0 con 0x7fbcbc13f650 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3367146363 0 0) 0x7fbcbc06f3d0 con 0x7fbcbc13f650 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcbc06f640 con 0x7fbcbc13f650 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2608766599 0 0) 0x7fbc9c004eb0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcbc06f3d0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbcb4001ef0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:18.458 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 10 monclient(hunting): dump: 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:18.459 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.462+0000 7fbcaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbcb00018f0 con 0x7fbcbc13f650 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient(hunting): dump: 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:18.460 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754926776 0 0) 0x7fbcbc06f3d0 con 0x7fbcbc1457f0 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 >> v1:172.21.15.79:6790/0 conn(0x7fbcbc13f650 legacy=0x7fbcbc137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 >> v1:172.21.15.79:6789/0 conn(0x7fbcbc0a9d70 legacy=0x7fbcbc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 1 monclient: found mon.b 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc06f920 con 0x7fbcbc1457f0 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:18.461 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.463+0000 7fbcaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 10 client.?.objecter ms_handle_connect 0x7fbcbc0a9d70 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 5 monclient: authenticate success, global_id 4501 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 10 monclient: _renew_subs 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcbc137f20 con 0x7fbcbc1457f0 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:18.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 10 monclient: _renew_subs 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcbc06e730 con 0x7fbcbc1457f0 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcc0e0f640 1 librados: init done 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcb4004f70 con 0x7fbcbc1457f0 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fbcb4005280 con 0x7fbcbc1457f0 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcaaffd640 10 monclient: dump: 2024-03-29T10:57:18.463 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.464+0000 7fbcb8d99640 4 set_mon_vals no callback set 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.465+0000 7fbcc0e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:18.464 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.465+0000 7fbcc0e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.465+0000 7fbcc0e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.465+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca0005080 con 0x7fbcbc1457f0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.469+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fbcb401b250 con 0x7fbcbc1457f0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.470+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7fbcb4084e50 con 0x7fbcbc1457f0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.470+0000 7fbcaaffd640 10 client.4501.objecter ms_dispatch 0x7fbcbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.470+0000 7fbcaaffd640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.470+0000 7fbcaaffd640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.470+0000 7fbcaaffd640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.470+0000 7fbcba59c640 10 monclient: get_auth_request con 0x7fbc9c06de20 auth_method 0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.471+0000 7fbcaaffd640 10 client.4501.objecter ms_handle_connect 0x7fbc9c06de20 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.473+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbcb4050730 con 0x7fbcbc1457f0 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.473+0000 7fbcaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:18.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.473+0000 7fbcaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:18.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.794+0000 7fbcc0e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:18.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.794+0000 7fbcc0e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:18.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.794+0000 7fbcc0e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:18.775 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.794+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbca0005330 con 0x7fbcbc1457f0 2024-03-29T10:57:18.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.796+0000 7fbcaaffd640 1 -- 172.21.15.79:0/667969197 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7fbcb4052660 con 0x7fbcbc1457f0 2024-03-29T10:57:18.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.796+0000 7fbcaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:18.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.796+0000 7fbcaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:18.777 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:18.777 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.799+0000 7fbcc0e0f640 10 librados: watch_flush enter 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.799+0000 7fbcc0e0f640 10 librados: watch_flush exit 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.799+0000 7fbcc0e0f640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.799+0000 7fbcc0e0f640 10 client.4501.objecter successfully canceled tick 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.801+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 >> v1:172.21.15.79:6816/4290694627 conn(0x7fbc9c06de20 legacy=0x7fbc9c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.801+0000 7fbcc0e0f640 10 monclient: shutdown 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.801+0000 7fbcc0e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:18.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.801+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 >> v1:172.21.15.82:6789/0 conn(0x7fbcbc1457f0 legacy=0x7fbcbc137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:18.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.802+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 shutdown_connections 2024-03-29T10:57:18.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.802+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 >> 172.21.15.79:0/667969197 conn(0x7fbcbc066a00 msgr2=0x7fbcbc139020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:18.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.802+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 shutdown_connections 2024-03-29T10:57:18.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.802+0000 7fbcc0e0f640 1 -- 172.21.15.79:0/667969197 wait complete. 2024-03-29T10:57:18.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:18.802+0000 7fbcc0e0f640 1 librados: shutdown 2024-03-29T10:57:18.804 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:25.611 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:25.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.788+0000 7feb150e6640 10 monclient: get_monmap_and_config 2024-03-29T10:57:25.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.788+0000 7feb150e6640 10 monclient: build_initial_monmap 2024-03-29T10:57:25.770 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.788+0000 7feb150e6640 10 monclient: monmap: 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:25.789751+0000 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:25.789751+0000 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:25.771 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:25.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.790+0000 7feb150e6640 1 Processor -- start 2024-03-29T10:57:25.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.790+0000 7feb150e6640 1 -- start start 2024-03-29T10:57:25.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.790+0000 7feb150e6640 10 monclient: init 2024-03-29T10:57:25.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:25.772 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:25.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient(hunting): picked mon.noname-b con 0x7feb100a9930 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:25.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient(hunting): picked mon.noname-c con 0x7feb10149dc0 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:25.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient(hunting): picked mon.noname-a con 0x7feb1014ff40 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:25.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100aae60 con 0x7feb1014ff40 2024-03-29T10:57:25.776 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100b4310 con 0x7feb10149dc0 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100afdd0 con 0x7feb100a9930 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.791+0000 7feb150e6640 10 monclient(hunting): authenticate will time out at 1565.083486s 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.794+0000 7feb0f59e640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7feb1014ff40 0x7feb10152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:46875/0 (socket says 172.21.15.79:46875) 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.794+0000 7feb0f59e640 1 -- 172.21.15.79:0/1060002066 learned_addr learned my addr 172.21.15.79:0/1060002066 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.795+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933876497 0 0) 0x7feb100aae60 con 0x7feb1014ff40 2024-03-29T10:57:25.777 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.795+0000 7feb0dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.795+0000 7feb0dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.795+0000 7feb0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.795+0000 7feb0dd9b640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.795+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf8003520 con 0x7feb1014ff40 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963657507 0 0) 0x7feb100afdd0 con 0x7feb100a9930 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:25.782 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb100aae60 con 0x7feb100a9930 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2413259591 0 0) 0x7feaf8003520 con 0x7feb1014ff40 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 >> v1:172.21.15.79:6790/0 conn(0x7feb100a9930 legacy=0x7feb10143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 >> v1:172.21.15.82:6789/0 conn(0x7feb10149dc0 legacy=0x7feb1014a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 monclient: found mon.noname-a 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1014a950 con 0x7feb1014ff40 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:57:25.783 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.797+0000 7feb0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb150e6640 5 monclient: authenticate success, global_id 4557 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb150e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7feb08001990 con 0x7feb1014ff40 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb0dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb0dd9b640 10 monclient: dump: 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:25.784 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.798+0000 7feb150e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb08001bb0 con 0x7feb1014ff40 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb0dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb0dd9b640 1 -- 172.21.15.79:0/1060002066 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7feb08001f40 con 0x7feb1014ff40 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb0dd9b640 10 monclient: dump: 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.785 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb150e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.799+0000 7feb150e6640 4 set_mon_vals no callback set 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.800+0000 7feb150e6640 10 monclient: shutdown 2024-03-29T10:57:25.786 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.800+0000 7feb150e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.800+0000 7feb150e6640 1 -- 172.21.15.79:0/1060002066 >> v1:172.21.15.79:6789/0 conn(0x7feb1014ff40 legacy=0x7feb10152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.800+0000 7feb150e6640 1 -- 172.21.15.79:0/1060002066 shutdown_connections 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.800+0000 7feb150e6640 1 -- 172.21.15.79:0/1060002066 >> 172.21.15.79:0/1060002066 conn(0x7feb100b35e0 msgr2=0x7feb100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.802+0000 7feb150e6640 1 -- 172.21.15.79:0/1060002066 shutdown_connections 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.804+0000 7feb150e6640 1 -- 172.21.15.79:0/1060002066 wait complete. 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 10 monclient: build_initial_monmap 2024-03-29T10:57:25.787 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 10 monclient: monmap: 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 1 librados: starting msgr at 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 1 librados: starting objecter 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 1 Processor -- start 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 1 -- start start 2024-03-29T10:57:25.788 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 1 librados: setting wanted keys 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 1 librados: calling monclient init 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.806+0000 7feb150e6640 10 monclient: init 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient(hunting): picked mon.noname-b con 0x7feb100a9930 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient(hunting): picked mon.noname-c con 0x7feb10149dc0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:25.789 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient(hunting): picked mon.noname-a con 0x7feb1014ff40 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:25.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10135fd0 con 0x7feb1014ff40 2024-03-29T10:57:25.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb101461b0 con 0x7feb100a9930 2024-03-29T10:57:25.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb101470a0 con 0x7feb10149dc0 2024-03-29T10:57:25.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:25.790 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.807+0000 7feb150e6640 10 monclient(hunting): authenticate will time out at 1565.099899s 2024-03-29T10:57:25.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.814+0000 7feb0f59e640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7feb1014ff40 0x7feb102000a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:47461/0 (socket says 172.21.15.79:47461) 2024-03-29T10:57:25.794 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.814+0000 7feb0f59e640 1 -- 172.21.15.79:0/158267280 learned_addr learned my addr 172.21.15.79:0/158267280 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:25.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.817+0000 7feaf77fe640 10 client.?.objecter ms_handle_connect 0x7feb1014ff40 2024-03-29T10:57:25.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:25.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 client.?.objecter ms_handle_connect 0x7feb100a9930 2024-03-29T10:57:25.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:25.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 client.?.objecter ms_handle_connect 0x7feb10149dc0 2024-03-29T10:57:25.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726999437 0 0) 0x7feb101461b0 con 0x7feb100a9930 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feafc004eb0 con 0x7feb100a9930 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1581441606 0 0) 0x7feafc004eb0 con 0x7feb100a9930 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb101461b0 con 0x7feb100a9930 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7feb00001930 con 0x7feb100a9930 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): dump: 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:25.801 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:25.802 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.818+0000 7feaf77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1006747427 0 0) 0x7feb101461b0 con 0x7feb100a9930 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 >> v1:172.21.15.79:6790/0 conn(0x7feb10149dc0 legacy=0x7feb10145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 >> v1:172.21.15.79:6789/0 conn(0x7feb1014ff40 legacy=0x7feb102000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 monclient: found mon.b 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:25.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10147380 con 0x7feb100a9930 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: _finish_auth 0 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb00003610 con 0x7feb100a9930 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7feb00003920 con 0x7feb100a9930 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feaf77fe640 10 monclient: dump: 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:25.804 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.819+0000 7feb0d59a640 4 set_mon_vals no callback set 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 5 monclient: authenticate success, global_id 4504 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 10 monclient: _renew_subs 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb10144510 con 0x7feb100a9930 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 10 monclient: _renew_subs 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1013dae0 con 0x7feb100a9930 2024-03-29T10:57:25.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.820+0000 7feb150e6640 1 librados: init done 2024-03-29T10:57:25.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.822+0000 7feb150e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:25.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.822+0000 7feb150e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:25.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.823+0000 7feb150e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:25.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.823+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb10075f10 con 0x7feb100a9930 2024-03-29T10:57:25.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.824+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7feb000198f0 con 0x7feb100a9930 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7feb00083880 con 0x7feb100a9930 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feb0e59c640 10 monclient: get_auth_request con 0x7feafc06ddb0 auth_method 0 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feaf77fe640 10 client.4504.objecter ms_dispatch 0x7feb10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feaf77fe640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feaf77fe640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feaf77fe640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T10:57:25.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.828+0000 7feaf77fe640 10 client.4504.objecter ms_handle_connect 0x7feafc06ddb0 2024-03-29T10:57:25.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.830+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb0004ef90 con 0x7feb100a9930 2024-03-29T10:57:25.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.830+0000 7feaf77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:25.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:25.830+0000 7feaf77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:26.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.182+0000 7feb150e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.182+0000 7feb150e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.182+0000 7feb150e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.182+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb10145650 con 0x7feb100a9930 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.183+0000 7feaf77fe640 1 -- 172.21.15.79:0/158267280 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7feb00051090 con 0x7feb100a9930 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.183+0000 7feaf77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.183+0000 7feaf77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:26.165 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.187+0000 7feb150e6640 10 librados: watch_flush enter 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.187+0000 7feb150e6640 10 librados: watch_flush exit 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.187+0000 7feb150e6640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.187+0000 7feb150e6640 10 client.4504.objecter successfully canceled tick 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.187+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 >> v1:172.21.15.79:6816/4290694627 conn(0x7feafc06ddb0 legacy=0x7feafc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.187+0000 7feb150e6640 10 monclient: shutdown 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.188+0000 7feb150e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:26.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.188+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 >> v1:172.21.15.82:6789/0 conn(0x7feb100a9930 legacy=0x7feb101358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:26.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.188+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 shutdown_connections 2024-03-29T10:57:26.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.188+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 >> 172.21.15.79:0/158267280 conn(0x7feb100b35e0 msgr2=0x7feb100b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:26.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.188+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 shutdown_connections 2024-03-29T10:57:26.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.188+0000 7feb150e6640 1 -- 172.21.15.79:0/158267280 wait complete. 2024-03-29T10:57:26.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:26.189+0000 7feb150e6640 1 librados: shutdown 2024-03-29T10:57:26.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:32.998 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:33.161 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.180+0000 7f389c8fc640 10 monclient: get_monmap_and_config 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.180+0000 7f389c8fc640 10 monclient: build_initial_monmap 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.180+0000 7f389c8fc640 10 monclient: monmap: 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:33.181671+0000 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:33.181671+0000 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:33.162 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:33.163 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:33.163 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:33.163 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:33.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.182+0000 7f389c8fc640 1 Processor -- start 2024-03-29T10:57:33.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.182+0000 7f389c8fc640 1 -- start start 2024-03-29T10:57:33.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.182+0000 7f389c8fc640 10 monclient: init 2024-03-29T10:57:33.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:33.164 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3890006410 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient(hunting): picked mon.noname-c con 0x7f389002a640 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f38977fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f3890006410 0x7f38900c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:44834/0 (socket says 172.21.15.79:44834) 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f38977fe640 1 -- 172.21.15.79:0/48843958 learned_addr learned my addr 172.21.15.79:0/48843958 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient(hunting): picked mon.noname-b con 0x7f38900327d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389002c820 con 0x7f3890006410 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389002a120 con 0x7f389002a640 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900c75b0 con 0x7f38900327d0 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:33.165 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.183+0000 7f389c8fc640 10 monclient(hunting): authenticate will time out at 1572.475946s 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202967406 0 0) 0x7f38900c75b0 con 0x7f38900327d0 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3888003520 con 0x7f38900327d0 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752897800 0 0) 0x7f389002c820 con 0x7f3890006410 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:33.166 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:33.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:33.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:57:33.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.184+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38900c75b0 con 0x7f3890006410 2024-03-29T10:57:33.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.185+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865100877 0 0) 0x7f38900c75b0 con 0x7f3890006410 2024-03-29T10:57:33.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.185+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 >> v1:172.21.15.79:6790/0 conn(0x7f38900327d0 legacy=0x7f38900d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.167 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.185+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 >> v1:172.21.15.82:6789/0 conn(0x7f389002a640 legacy=0x7f38900320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 1 monclient: found mon.noname-a 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3890033110 con 0x7f3890006410 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f388c001670 con 0x7f3890006410 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:33.168 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f38967fc640 10 monclient: dump: 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f389c8fc640 5 monclient: authenticate success, global_id 4560 2024-03-29T10:57:33.169 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.187+0000 7f389c8fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:33.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f388c0017d0 con 0x7f3890006410 2024-03-29T10:57:33.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f38967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:33.170 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f38967fc640 1 -- 172.21.15.79:0/48843958 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f388c001be0 con 0x7f3890006410 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f38967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f38967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f38967fc640 10 monclient: dump: 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f389c8fc640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:33.171 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.188+0000 7f389c8fc640 4 set_mon_vals no callback set 2024-03-29T10:57:33.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.189+0000 7f389c8fc640 10 monclient: shutdown 2024-03-29T10:57:33.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.189+0000 7f389c8fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:33.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.189+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 >> v1:172.21.15.79:6789/0 conn(0x7f3890006410 legacy=0x7f38900c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.190+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 shutdown_connections 2024-03-29T10:57:33.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.190+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 >> 172.21.15.79:0/48843958 conn(0x7f38900058f0 msgr2=0x7f3890005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:33.172 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.191+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 shutdown_connections 2024-03-29T10:57:33.173 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.191+0000 7f389c8fc640 1 -- 172.21.15.79:0/48843958 wait complete. 2024-03-29T10:57:33.173 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.192+0000 7f389c8fc640 10 monclient: build_initial_monmap 2024-03-29T10:57:33.173 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.192+0000 7f389c8fc640 10 monclient: monmap: 2024-03-29T10:57:33.173 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:33.173 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:33.173 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.192+0000 7f389c8fc640 1 librados: starting msgr at 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.192+0000 7f389c8fc640 1 librados: starting objecter 2024-03-29T10:57:33.174 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.193+0000 7f389c8fc640 1 Processor -- start 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.194+0000 7f389c8fc640 1 -- start start 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.194+0000 7f389c8fc640 1 librados: setting wanted keys 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.194+0000 7f389c8fc640 1 librados: calling monclient init 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.194+0000 7f389c8fc640 10 monclient: init 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3890006410 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:33.177 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient(hunting): picked mon.noname-c con 0x7f389002a640 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:33.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient(hunting): picked mon.noname-a con 0x7f38900327d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:33.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900c3f90 con 0x7f38900327d0 2024-03-29T10:57:33.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900c2c90 con 0x7f3890006410 2024-03-29T10:57:33.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900c2e20 con 0x7f389002a640 2024-03-29T10:57:33.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:33.178 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.195+0000 7f389c8fc640 10 monclient(hunting): authenticate will time out at 1572.488045s 2024-03-29T10:57:33.179 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.197+0000 7f3897fff640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f38900327d0 0x7f3890173f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:44434/0 (socket says 172.21.15.79:44434) 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.197+0000 7f3897fff640 1 -- 172.21.15.79:0/3959624970 learned_addr learned my addr 172.21.15.79:0/3959624970 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.197+0000 7f3883fff640 10 client.?.objecter ms_handle_connect 0x7f38900327d0 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.197+0000 7f3883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.198+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342497301 0 0) 0x7f38900c3f90 con 0x7f38900327d0 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.198+0000 7f3883fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.198+0000 7f3883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.198+0000 7f3883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.198+0000 7f3883fff640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.198+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f387c004eb0 con 0x7f38900327d0 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.199+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2739171453 0 0) 0x7f387c004eb0 con 0x7f38900327d0 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.199+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38900c3f90 con 0x7f38900327d0 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.199+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f38840018f0 con 0x7f38900327d0 2024-03-29T10:57:33.180 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.199+0000 7f3883fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient(hunting): dump: 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.182 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 client.?.objecter ms_handle_connect 0x7f3890006410 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 client.?.objecter ms_handle_connect 0x7f389002a640 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1153589207 0 0) 0x7f38900c3f90 con 0x7f38900327d0 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6790/0 conn(0x7f389002a640 legacy=0x7f3890169df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.82:6789/0 conn(0x7f3890006410 legacy=0x7f38901696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:33.183 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 1 monclient: found mon.a 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38900c2fb0 con 0x7f38900327d0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f3883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 10 monclient: _renew_subs 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389014eab0 con 0x7f38900327d0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 10 monclient: _renew_subs 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38900c3980 con 0x7f38900327d0 2024-03-29T10:57:33.184 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.201+0000 7f389c8fc640 1 librados: init done 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3897fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6789/0 conn(0x7f38900327d0 legacy=0x7f3890173f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3897fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6789/0 conn(0x7f38900327d0 legacy=0x7f3890173f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3897fff640 1 --1- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6789/0 conn(0x7f38900327d0 0x7f3890173f90 :-1 s=OPENED pgs=962521 cs=1 l=1).handle_message read tag failed 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3897fff640 1 --1- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6789/0 conn(0x7f38900327d0 0x7f3890173f90 :-1 s=OPENED pgs=962521 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3883fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.79:6789/0 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3883fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6789/0 conn(0x7f38900327d0 legacy=0x7f3890173f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3883fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:33.185 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.202+0000 7f3883fff640 10 monclient(hunting): picked mon.a con 0x7f387c006150 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 10 monclient(hunting): picked mon.b con 0x7f3890006410 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 10 monclient(hunting): picked mon.c con 0x7f389002a640 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f38900c3f90 con 0x7f387c006150 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f38900c2c90 con 0x7f3890006410 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f38900c2e20 con 0x7f389002a640 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 10 client.4563.objecter ms_handle_connect 0x7f389002a640 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.203+0000 7f3883fff640 10 client.4563.objecter resend_mon_ops 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473030403 0 0) 0x7f38900c2e20 con 0x7f389002a640 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f387c013880 con 0x7f389002a640 2024-03-29T10:57:33.186 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 10 client.4563.objecter ms_handle_connect 0x7f387c006150 2024-03-29T10:57:33.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.204+0000 7f3883fff640 10 client.4563.objecter resend_mon_ops 2024-03-29T10:57:33.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f389c8fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:33.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f389c8fc640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:33.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618970874 0 0) 0x7f38900c3f90 con 0x7f387c006150 2024-03-29T10:57:33.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:33.187 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:33.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:57:33.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f38900c2e20 con 0x7f387c006150 2024-03-29T10:57:33.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1485637420 0 0) 0x7f38900c2e20 con 0x7f387c006150 2024-03-29T10:57:33.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6790/0 conn(0x7f389002a640 legacy=0x7f387c00fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.188 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.205+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.82:6789/0 conn(0x7f3890006410 legacy=0x7f387c00c310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:33.189 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:33.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 1 monclient: found mon.a 2024-03-29T10:57:33.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:33.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f387c004eb0 con 0x7f387c006150 2024-03-29T10:57:33.190 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389014ee00 con 0x7f387c006150 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38900c2c90 con 0x7f387c006150 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f388c0015f0 con 0x7f387c006150 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.209+0000 7f3883fff640 10 monclient: dump: 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.191 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:33.192 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:33.192 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:33.192 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:33.192 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:33.192 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:33.192 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:33.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.211+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f388c001940 con 0x7f387c006150 2024-03-29T10:57:33.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.211+0000 7f3883fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:33.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.211+0000 7f3895ffb640 4 set_mon_vals no callback set 2024-03-29T10:57:33.193 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.212+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f388c001b30 con 0x7f387c006150 2024-03-29T10:57:33.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.213+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f388c084180 con 0x7f387c006150 2024-03-29T10:57:33.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.213+0000 7f3883fff640 10 client.4563.objecter ms_dispatch 0x7f3898000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:33.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.213+0000 7f3896ffd640 10 monclient: get_auth_request con 0x7f38900327d0 auth_method 0 2024-03-29T10:57:33.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.213+0000 7f3883fff640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:33.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.213+0000 7f3883fff640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:33.194 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.214+0000 7f3883fff640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T10:57:33.195 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.215+0000 7f3883fff640 10 client.4563.objecter ms_handle_connect 0x7f38900327d0 2024-03-29T10:57:33.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.221+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f388c04f890 con 0x7f387c006150 2024-03-29T10:57:33.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.221+0000 7f3883fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:33.202 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.221+0000 7f3883fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:33.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.231+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f388c051990 con 0x7f387c006150 2024-03-29T10:57:33.211 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.231+0000 7f3883fff640 10 monclient: handle_mon_command_ack 1 not found 2024-03-29T10:57:33.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.582+0000 7f389c8fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:33.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.582+0000 7f389c8fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:33.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.582+0000 7f389c8fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:33.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.582+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3890004cd0 con 0x7f387c006150 2024-03-29T10:57:33.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.584+0000 7f3883fff640 1 -- 172.21.15.79:0/3959624970 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f388c0531b0 con 0x7f387c006150 2024-03-29T10:57:33.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.584+0000 7f3883fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:33.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.584+0000 7f3883fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:33.565 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:33.565 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:33.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.587+0000 7f389c8fc640 10 librados: watch_flush enter 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.587+0000 7f389c8fc640 10 librados: watch_flush exit 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.587+0000 7f389c8fc640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.587+0000 7f389c8fc640 10 client.4563.objecter successfully canceled tick 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.587+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6816/4290694627 conn(0x7f38900327d0 legacy=0x7f387c07b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 10 monclient: shutdown 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:33.571 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 >> v1:172.21.15.79:6789/0 conn(0x7f387c006150 legacy=0x7f387c006530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:33.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 shutdown_connections 2024-03-29T10:57:33.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 >> 172.21.15.79:0/3959624970 conn(0x7f38900058f0 msgr2=0x7f38900043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:33.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 shutdown_connections 2024-03-29T10:57:33.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 1 -- 172.21.15.79:0/3959624970 wait complete. 2024-03-29T10:57:33.572 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:33.588+0000 7f389c8fc640 1 librados: shutdown 2024-03-29T10:57:33.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:40.395 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:40.545 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.565+0000 7f1460c0f640 10 monclient: get_monmap_and_config 2024-03-29T10:57:40.545 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.565+0000 7f1460c0f640 10 monclient: build_initial_monmap 2024-03-29T10:57:40.545 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.565+0000 7f1460c0f640 10 monclient: monmap: 2024-03-29T10:57:40.545 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:40.545 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:40.566025+0000 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:40.566025+0000 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.546 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.566+0000 7f1460c0f640 1 Processor -- start 2024-03-29T10:57:40.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.566+0000 7f1460c0f640 1 -- start start 2024-03-29T10:57:40.547 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.566+0000 7f1460c0f640 10 monclient: init 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f145c1457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f145c13f650 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f145c0a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c0aaef0 con 0x7f145c1457f0 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c145bd0 con 0x7f145c13f650 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c145eb0 con 0x7f145c0a9d70 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f1460c0f640 10 monclient(hunting): authenticate will time out at 1579.859723s 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f145ad9d640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f145c0a9d70 0x7f145c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:42762/0 (socket says 172.21.15.79:42762) 2024-03-29T10:57:40.548 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.567+0000 7f145ad9d640 1 -- 172.21.15.79:0/142455641 learned_addr learned my addr 172.21.15.79:0/142455641 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63698986 0 0) 0x7f145c145bd0 con 0x7f145c13f650 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448003520 con 0x7f145c13f650 2024-03-29T10:57:40.550 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323054452 0 0) 0x7f145c0aaef0 con 0x7f145c1457f0 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f145c145bd0 con 0x7f145c1457f0 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346127851 0 0) 0x7f145c145eb0 con 0x7f145c0a9d70 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f145c0aaef0 con 0x7f145c0a9d70 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360236045 0 0) 0x7f1448003520 con 0x7f145c13f650 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 >> v1:172.21.15.79:6790/0 conn(0x7f145c0a9d70 legacy=0x7f145c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 >> v1:172.21.15.79:6789/0 conn(0x7f145c1457f0 legacy=0x7f145c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:40.551 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 monclient: found mon.noname-c 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c146340 con 0x7f145c13f650 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: _finish_auth 0 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f1460c0f640 5 monclient: authenticate success, global_id 4510 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f144c001690 con 0x7f145c13f650 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f1460c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f145959a640 10 monclient: dump: 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.552 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.568+0000 7f1460c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f144c0017f0 con 0x7f145c13f650 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f145959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f145959a640 1 -- 172.21.15.79:0/142455641 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f144c001c00 con 0x7f145c13f650 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f145959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f145959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f145959a640 10 monclient: dump: 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:40.553 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f1460c0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:40.554 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.569+0000 7f1460c0f640 4 set_mon_vals no callback set 2024-03-29T10:57:40.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.570+0000 7f1460c0f640 10 monclient: shutdown 2024-03-29T10:57:40.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.570+0000 7f1460c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:40.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.570+0000 7f1460c0f640 1 -- 172.21.15.79:0/142455641 >> v1:172.21.15.82:6789/0 conn(0x7f145c13f650 legacy=0x7f145c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.570+0000 7f1460c0f640 1 -- 172.21.15.79:0/142455641 shutdown_connections 2024-03-29T10:57:40.559 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.570+0000 7f1460c0f640 1 -- 172.21.15.79:0/142455641 >> 172.21.15.79:0/142455641 conn(0x7f145c066a00 msgr2=0x7f145c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.570+0000 7f1460c0f640 1 -- 172.21.15.79:0/142455641 shutdown_connections 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.571+0000 7f1460c0f640 1 -- 172.21.15.79:0/142455641 wait complete. 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 10 monclient: build_initial_monmap 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 10 monclient: monmap: 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 1 librados: starting msgr at 2024-03-29T10:57:40.560 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 1 librados: starting objecter 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 1 Processor -- start 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 1 -- start start 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 1 librados: setting wanted keys 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 1 librados: calling monclient init 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.572+0000 7f1460c0f640 10 monclient: init 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f145c0a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f145c13f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f145c1457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c1cb940 con 0x7f145c1457f0 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c1cc7d0 con 0x7f145c0a9d70 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c1ccab0 con 0x7f145c13f650 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:40.561 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f1460c0f640 10 monclient(hunting): authenticate will time out at 1579.866411s 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f145a59c640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f145c0a9d70 0x7f145c138cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:53456/0 (socket says 172.21.15.79:53456) 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.574+0000 7f145a59c640 1 -- 172.21.15.79:0/2158423675 learned_addr learned my addr 172.21.15.79:0/2158423675 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 client.?.objecter ms_handle_connect 0x7f145c0a9d70 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 client.?.objecter ms_handle_connect 0x7f145c1457f0 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 client.?.objecter ms_handle_connect 0x7f145c13f650 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47610023 0 0) 0x7f145c1ccab0 con 0x7f145c13f650 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:57:40.562 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f143c004eb0 con 0x7f145c13f650 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507973152 0 0) 0x7f145c1cb940 con 0x7f145c1457f0 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f145c1ccab0 con 0x7f145c1457f0 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897186090 0 0) 0x7f143c004eb0 con 0x7f145c13f650 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f145c1cb940 con 0x7f145c13f650 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.575+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f144c001b00 con 0x7f145c13f650 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:40.563 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3791517231 0 0) 0x7f145c1ccab0 con 0x7f145c1457f0 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f143c004eb0 con 0x7f145c1457f0 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1454001bf0 con 0x7f145c1457f0 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:40.564 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 254965451 0 0) 0x7f145c1cb940 con 0x7f145c13f650 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 >> v1:172.21.15.82:6789/0 conn(0x7f145c0a9d70 legacy=0x7f145c138cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 >> v1:172.21.15.79:6789/0 conn(0x7f145c1457f0 legacy=0x7f145c1e2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 monclient: found mon.c 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:40.565 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c1ccd90 con 0x7f145c13f650 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1460c0f640 5 monclient: authenticate success, global_id 4664 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1460c0f640 10 monclient: _renew_subs 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.576+0000 7f1460c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c1cbdd0 con 0x7f145c13f650 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1460c0f640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1460c0f640 10 monclient: _renew_subs 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1460c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c1395b0 con 0x7f145c13f650 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1460c0f640 1 librados: init done 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f144c0010a0 con 0x7f145c13f650 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1452ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:40.566 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1458d99640 4 set_mon_vals no callback set 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f144c0038a0 con 0x7f145c13f650 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1452ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1452ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.577+0000 7f1452ffd640 10 monclient: dump: 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.578+0000 7f1460c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.578+0000 7f1460c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:40.567 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.578+0000 7f1460c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:40.568 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.578+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1440005080 con 0x7f145c13f650 2024-03-29T10:57:40.568 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.580+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f144c0198f0 con 0x7f145c13f650 2024-03-29T10:57:40.568 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.580+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f144c083800 con 0x7f145c13f650 2024-03-29T10:57:40.568 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.580+0000 7f1452ffd640 10 client.4664.objecter ms_dispatch 0x7f145c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:40.568 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.580+0000 7f1452ffd640 3 client.4664.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.580+0000 7f1452ffd640 3 client.4664.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.581+0000 7f1452ffd640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.581+0000 7f145a59c640 10 monclient: get_auth_request con 0x7f143c06de70 auth_method 0 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.581+0000 7f1452ffd640 10 client.4664.objecter ms_handle_connect 0x7f143c06de70 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.587+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f144c04f140 con 0x7f145c13f650 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.587+0000 7f1452ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:40.569 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.587+0000 7f1452ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:40.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.963+0000 7f1460c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:40.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.963+0000 7f1460c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:40.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.963+0000 7f1460c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:57:40.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.963+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1440005330 con 0x7f145c13f650 2024-03-29T10:57:40.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.964+0000 7f1452ffd640 1 -- 172.21.15.79:0/2158423675 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f144c04f140 con 0x7f145c13f650 2024-03-29T10:57:40.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.964+0000 7f1452ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:40.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.964+0000 7f1452ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:40.944 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:40.945 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 10 librados: watch_flush enter 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 10 librados: watch_flush exit 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 10 client.4664.objecter successfully canceled tick 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 >> v1:172.21.15.79:6816/4290694627 conn(0x7f143c06de70 legacy=0x7f143c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 10 monclient: shutdown 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 >> v1:172.21.15.79:6790/0 conn(0x7f145c13f650 legacy=0x7f145c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:40.949 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 shutdown_connections 2024-03-29T10:57:40.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 >> 172.21.15.79:0/2158423675 conn(0x7f145c066a00 msgr2=0x7f145c13cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:40.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 shutdown_connections 2024-03-29T10:57:40.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 -- 172.21.15.79:0/2158423675 wait complete. 2024-03-29T10:57:40.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:40.968+0000 7f1460c0f640 1 librados: shutdown 2024-03-29T10:57:40.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:47.771 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.945+0000 7f5c3ca9a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.945+0000 7f5c3ca9a640 10 monclient: build_initial_monmap 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.945+0000 7f5c3ca9a640 10 monclient: monmap: 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:47.946347+0000 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:47.946347+0000 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:47.926 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:47.927 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:47.927 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.927 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.946+0000 7f5c3ca9a640 1 Processor -- start 2024-03-29T10:57:47.928 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.946+0000 7f5c3ca9a640 1 -- start start 2024-03-29T10:57:47.928 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.946+0000 7f5c3ca9a640 10 monclient: init 2024-03-29T10:57:47.928 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.947+0000 7f5c3ca9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:47.928 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.947+0000 7f5c3ca9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:47.929 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.947+0000 7f5c3ca9a640 10 monclient(hunting): picked mon.noname-b con 0x7f5c381457f0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3659c640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f5c381457f0 0x7f5c3813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:36352/0 (socket says 172.21.15.79:36352) 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3813f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 10 monclient(hunting): picked mon.noname-c con 0x7f5c380a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380aaef0 con 0x7f5c3813f650 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c38145bd0 con 0x7f5c380a9d70 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c38145eb0 con 0x7f5c381457f0 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3ca9a640 10 monclient(hunting): authenticate will time out at 1587.241501s 2024-03-29T10:57:47.939 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.949+0000 7f5c3659c640 1 -- 172.21.15.79:0/3694995108 learned_addr learned my addr 172.21.15.79:0/3694995108 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298589088 0 0) 0x7f5c38145eb0 con 0x7f5c381457f0 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c20003520 con 0x7f5c381457f0 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589690311 0 0) 0x7f5c380aaef0 con 0x7f5c3813f650 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.950+0000 7f5c3559a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.951+0000 7f5c3559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.951+0000 7f5c3559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.951+0000 7f5c3559a640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.951+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c38145eb0 con 0x7f5c3813f650 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.951+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1915945496 0 0) 0x7f5c20003520 con 0x7f5c381457f0 2024-03-29T10:57:47.940 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.951+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 >> v1:172.21.15.82:6789/0 conn(0x7f5c380a9d70 legacy=0x7f5c3814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 >> v1:172.21.15.79:6789/0 conn(0x7f5c3813f650 legacy=0x7f5c380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 1 monclient: found mon.noname-b 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c38146340 con 0x7f5c381457f0 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 10 monclient: _finish_auth 0 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5c30001a10 con 0x7f5c381457f0 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3ca9a640 5 monclient: authenticate success, global_id 4667 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.952+0000 7f5c3ca9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:47.941 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: dump: 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c30001b70 con 0x7f5c381457f0 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3ca9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:47.942 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 1 -- 172.21.15.79:0/3694995108 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5c30001fc0 con 0x7f5c381457f0 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3ca9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3ca9a640 4 set_mon_vals no callback set 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.953+0000 7f5c3559a640 10 monclient: dump: 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.943 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.954+0000 7f5c3ca9a640 10 monclient: shutdown 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.954+0000 7f5c3ca9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:47.944 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.954+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/3694995108 >> v1:172.21.15.79:6790/0 conn(0x7f5c381457f0 legacy=0x7f5c3813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:47.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.954+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/3694995108 shutdown_connections 2024-03-29T10:57:47.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.954+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/3694995108 >> 172.21.15.79:0/3694995108 conn(0x7f5c38066a00 msgr2=0x7f5c38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:47.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.955+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/3694995108 shutdown_connections 2024-03-29T10:57:47.945 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.955+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/3694995108 wait complete. 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.956+0000 7f5c3ca9a640 10 monclient: build_initial_monmap 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.956+0000 7f5c3ca9a640 10 monclient: monmap: 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:47.950 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 1 librados: starting msgr at 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 1 librados: starting objecter 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 1 Processor -- start 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 1 -- start start 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 1 librados: setting wanted keys 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 1 librados: calling monclient init 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.957+0000 7f5c3ca9a640 10 monclient: init 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.958+0000 7f5c3ca9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.958+0000 7f5c3ca9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.958+0000 7f5c3ca9a640 10 monclient(hunting): picked mon.noname-c con 0x7f5c380a9d70 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.958+0000 7f5c3ca9a640 10 monclient(hunting): picked mon.noname-a con 0x7f5c3813f650 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.958+0000 7f5c3659c640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f5c380a9d70 0x7f5c3813b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:37406/0 (socket says 172.21.15.79:37406) 2024-03-29T10:57:47.951 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3659c640 1 -- 172.21.15.79:0/95711244 learned_addr learned my addr 172.21.15.79:0/95711244 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3ca9a640 10 monclient(hunting): picked mon.noname-b con 0x7f5c381457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c381359f0 con 0x7f5c3813f650 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3813ace0 con 0x7f5c381457f0 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3813a8c0 con 0x7f5c380a9d70 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3ca9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c3ca9a640 10 monclient(hunting): authenticate will time out at 1587.251692s 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c26ffd640 10 client.?.objecter ms_handle_connect 0x7f5c380a9d70 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.959+0000 7f5c26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.961+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400230265 0 0) 0x7f5c3813a8c0 con 0x7f5c380a9d70 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.961+0000 7f5c26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.961+0000 7f5c26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.961+0000 7f5c26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.961+0000 7f5c26ffd640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:57:47.952 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.961+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18004eb0 con 0x7f5c380a9d70 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486617845 0 0) 0x7f5c18004eb0 con 0x7f5c380a9d70 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c3813a8c0 con 0x7f5c380a9d70 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5c30001f40 con 0x7f5c380a9d70 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:47.953 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 10 client.?.objecter ms_handle_connect 0x7f5c381457f0 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.962+0000 7f5c26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 client.?.objecter ms_handle_connect 0x7f5c3813f650 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153536162 0 0) 0x7f5c2c001210 con 0x7f5c3813f650 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18004eb0 con 0x7f5c3813f650 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522421437 0 0) 0x7f5c3813ace0 con 0x7f5c381457f0 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:47.954 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c001210 con 0x7f5c381457f0 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129481403 0 0) 0x7f5c18004eb0 con 0x7f5c3813f650 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c18005b70 con 0x7f5c3813f650 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5c2c0018b0 con 0x7f5c3813f650 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.955 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3491903191 0 0) 0x7f5c18005b70 con 0x7f5c3813f650 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.963+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 >> v1:172.21.15.79:6790/0 conn(0x7f5c380a9d70 legacy=0x7f5c3813b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 >> v1:172.21.15.82:6789/0 conn(0x7f5c381457f0 legacy=0x7f5c381e2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 1 monclient: found mon.a 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3813b8d0 con 0x7f5c3813f650 2024-03-29T10:57:47.956 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 5 monclient: authenticate success, global_id 4575 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 10 monclient: _renew_subs 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c381396f0 con 0x7f5c3813f650 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 10 monclient: _renew_subs 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c381d7e70 con 0x7f5c3813f650 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c3ca9a640 1 librados: init done 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c2c0035f0 con 0x7f5c3813f650 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5c2c0018b0 con 0x7f5c3813f650 2024-03-29T10:57:47.957 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c26ffd640 10 monclient: dump: 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:47.958 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.964+0000 7f5c34d99640 4 set_mon_vals no callback set 2024-03-29T10:57:47.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.966+0000 7f5c3ca9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:47.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.966+0000 7f5c3ca9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:47.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.966+0000 7f5c3ca9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:47.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.966+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c005080 con 0x7f5c3813f650 2024-03-29T10:57:47.959 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.966+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f5c2c0195d0 con 0x7f5c3813f650 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.967+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2563033506 0 0) 0x7f5c2c0530a0 con 0x7f5c3813f650 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.967+0000 7f5c26ffd640 10 client.4575.objecter ms_dispatch 0x7f5c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.967+0000 7f5c26ffd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.967+0000 7f5c26ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.967+0000 7f5c26ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.968+0000 7f5c3659c640 10 monclient: get_auth_request con 0x7f5c1806e290 auth_method 0 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.968+0000 7f5c26ffd640 10 client.4575.objecter ms_handle_connect 0x7f5c1806e290 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.977+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c2c04f340 con 0x7f5c3813f650 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.977+0000 7f5c26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:47.960 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:47.977+0000 7f5c26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:48.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.306+0000 7f5c3ca9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:48.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.306+0000 7f5c3ca9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:48.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.306+0000 7f5c3ca9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:57:48.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.306+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c1c005330 con 0x7f5c3813f650 2024-03-29T10:57:48.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.309+0000 7f5c26ffd640 1 -- 172.21.15.79:0/95711244 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1428579564) 0x7f5c2c04f340 con 0x7f5c3813f650 2024-03-29T10:57:48.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.309+0000 7f5c26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:48.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.309+0000 7f5c26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:48.290 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:48.290 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":10,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:47:51.190686+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:47:51.190686+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 10 librados: watch_flush enter 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 10 librados: watch_flush exit 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 10 client.4575.objecter successfully canceled tick 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 >> v1:172.21.15.79:6816/4290694627 conn(0x7f5c1806e290 legacy=0x7f5c18070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 10 monclient: shutdown 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:48.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 >> v1:172.21.15.79:6789/0 conn(0x7f5c3813f650 legacy=0x7f5c381d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:48.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.313+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 shutdown_connections 2024-03-29T10:57:48.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.314+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 >> 172.21.15.79:0/95711244 conn(0x7f5c38066a00 msgr2=0x7f5c380b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:48.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.314+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 shutdown_connections 2024-03-29T10:57:48.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.314+0000 7f5c3ca9a640 1 -- 172.21.15.79:0/95711244 wait complete. 2024-03-29T10:57:48.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:48.314+0000 7f5c3ca9a640 1 librados: shutdown 2024-03-29T10:57:48.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:55.118 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:55.272 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.291+0000 7f004364a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:55.272 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.291+0000 7f004364a640 10 monclient: build_initial_monmap 2024-03-29T10:57:55.272 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.291+0000 7f004364a640 10 monclient: monmap: 2024-03-29T10:57:55.272 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:55.272 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:55.272 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:57:55.292452+0000 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:57:55.292452+0000 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:57:55.273 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:55.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.293+0000 7f004364a640 1 Processor -- start 2024-03-29T10:57:55.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.293+0000 7f004364a640 1 -- start start 2024-03-29T10:57:55.274 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.293+0000 7f004364a640 10 monclient: init 2024-03-29T10:57:55.276 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:55.276 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:55.276 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient(hunting): picked mon.noname-a con 0x7f003c0af2e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:55.276 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient(hunting): picked mon.noname-b con 0x7f003c06d8c0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:55.277 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient(hunting): picked mon.noname-c con 0x7f003c1417e0 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:55.277 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0af770 con 0x7f003c0af2e0 2024-03-29T10:57:55.277 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0b00c0 con 0x7f003c1417e0 2024-03-29T10:57:55.277 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c13bbf0 con 0x7f003c06d8c0 2024-03-29T10:57:55.277 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:55.277 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.294+0000 7f004364a640 10 monclient(hunting): authenticate will time out at 1594.586786s 2024-03-29T10:57:55.278 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.297+0000 7f0041be7640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f003c1417e0 0x7f003c141c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:33223/0 (socket says 172.21.15.79:33223) 2024-03-29T10:57:55.278 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.297+0000 7f0041be7640 1 -- 172.21.15.79:0/2211247022 learned_addr learned my addr 172.21.15.79:0/2211247022 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:55.278 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.298+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962986185 0 0) 0x7f003c0b00c0 con 0x7f003c1417e0 2024-03-29T10:57:55.278 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.298+0000 7f0033fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:55.279 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.298+0000 7f0033fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:55.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:55.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:57:55.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0028003520 con 0x7f003c1417e0 2024-03-29T10:57:55.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888117510 0 0) 0x7f003c0af770 con 0x7f003c0af2e0 2024-03-29T10:57:55.283 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c0b00c0 con 0x7f003c0af2e0 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.302+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885069455 0 0) 0x7f003c13bbf0 con 0x7f003c06d8c0 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:57:55.284 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c0af770 con 0x7f003c06d8c0 2024-03-29T10:57:55.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178277396 0 0) 0x7f003c0af770 con 0x7f003c06d8c0 2024-03-29T10:57:55.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 >> v1:172.21.15.82:6789/0 conn(0x7f003c1417e0 legacy=0x7f003c141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 >> v1:172.21.15.79:6789/0 conn(0x7f003c0af2e0 legacy=0x7f003c0a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.303+0000 7f0033fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:55.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 1 monclient: found mon.noname-b 2024-03-29T10:57:55.287 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c1458d0 con 0x7f003c06d8c0 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0034001860 con 0x7f003c06d8c0 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f0033fff640 10 monclient: dump: 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:55.288 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f004364a640 5 monclient: authenticate success, global_id 4673 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.304+0000 7f004364a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0034001a40 con 0x7f003c06d8c0 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f0033fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f0033fff640 1 -- 172.21.15.79:0/2211247022 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0034001e90 con 0x7f003c06d8c0 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f0033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f0033fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f0033fff640 10 monclient: dump: 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:55.289 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f004364a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.305+0000 7f004364a640 4 set_mon_vals no callback set 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.306+0000 7f004364a640 10 monclient: shutdown 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.306+0000 7f004364a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.306+0000 7f004364a640 1 -- 172.21.15.79:0/2211247022 >> v1:172.21.15.79:6790/0 conn(0x7f003c06d8c0 legacy=0x7f003c13b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.306+0000 7f004364a640 1 -- 172.21.15.79:0/2211247022 shutdown_connections 2024-03-29T10:57:55.290 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.306+0000 7f004364a640 1 -- 172.21.15.79:0/2211247022 >> 172.21.15.79:0/2211247022 conn(0x7f003c0b3a00 msgr2=0x7f003c0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:55.291 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.311+0000 7f004364a640 1 -- 172.21.15.79:0/2211247022 shutdown_connections 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.311+0000 7f004364a640 1 -- 172.21.15.79:0/2211247022 wait complete. 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.312+0000 7f004364a640 10 monclient: build_initial_monmap 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.312+0000 7f004364a640 10 monclient: monmap: 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:57:55.293 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 1 librados: starting msgr at 2024-03-29T10:57:55.294 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 1 librados: starting objecter 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 1 Processor -- start 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 1 -- start start 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 1 librados: setting wanted keys 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 1 librados: calling monclient init 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.313+0000 7f004364a640 10 monclient: init 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.314+0000 7f004364a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:55.295 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.314+0000 7f004364a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:55.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.315+0000 7f004364a640 10 monclient(hunting): picked mon.noname-b con 0x7f003c06d8c0 addr v1:172.21.15.82:6789/0 2024-03-29T10:57:55.297 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 10 monclient(hunting): picked mon.noname-c con 0x7f003c1417e0 addr v1:172.21.15.79:6790/0 2024-03-29T10:57:55.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 10 monclient(hunting): picked mon.noname-a con 0x7f003c13c9a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:57:55.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c13d480 con 0x7f003c13c9a0 2024-03-29T10:57:55.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c12a570 con 0x7f003c06d8c0 2024-03-29T10:57:55.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c13deb0 con 0x7f003c1417e0 2024-03-29T10:57:55.302 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:55.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.317+0000 7f004364a640 10 monclient(hunting): authenticate will time out at 1594.609409s 2024-03-29T10:57:55.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.320+0000 7f00413e6640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f003c06d8c0 0x7f003c135af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:33329/0 (socket says 172.21.15.79:33329) 2024-03-29T10:57:55.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.320+0000 7f00413e6640 1 -- 172.21.15.79:0/4188071482 learned_addr learned my addr 172.21.15.79:0/4188071482 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:57:55.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.321+0000 7f0031ffb640 10 client.?.objecter ms_handle_connect 0x7f003c06d8c0 2024-03-29T10:57:55.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.321+0000 7f0031ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:55.303 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 10 client.?.objecter ms_handle_connect 0x7f003c1417e0 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569243334 0 0) 0x7f003c12a570 con 0x7f003c06d8c0 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 10 monclient(hunting): my global_id is 4519 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.322+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020004eb0 con 0x7f003c06d8c0 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547938811 0 0) 0x7f0020004eb0 con 0x7f003c06d8c0 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f003c12a570 con 0x7f003c06d8c0 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0038001990 con 0x7f003c06d8c0 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient(hunting): dump: 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:55.304 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:55.305 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3741276723 0 0) 0x7f003c12a570 con 0x7f003c06d8c0 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 >> v1:172.21.15.79:6790/0 conn(0x7f003c1417e0 legacy=0x7f003c1ded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 >> v1:172.21.15.79:6789/0 conn(0x7f003c13c9a0 legacy=0x7f003c1e24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 monclient: found mon.b 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c13e190 con 0x7f003c06d8c0 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.323+0000 7f0031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:55.306 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 5 monclient: authenticate success, global_id 4519 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 10 monclient: _renew_subs 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c136200 con 0x7f003c06d8c0 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0038001c70 con 0x7f003c06d8c0 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f0031ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 10 monclient: _renew_subs 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c13cf30 con 0x7f003c06d8c0 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f004364a640 1 librados: init done 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f0038001990 con 0x7f003c06d8c0 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f0031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f0031ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.324+0000 7f0031ffb640 10 monclient: dump: 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:57:55.307 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.325+0000 7f004364a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.326+0000 7f004364a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:55.308 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.326+0000 7f004364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:55.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.326+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c04f730 con 0x7f003c06d8c0 2024-03-29T10:57:55.309 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.326+0000 7f00337fe640 4 set_mon_vals no callback set 2024-03-29T10:57:55.310 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.330+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f0038003940 con 0x7f003c06d8c0 2024-03-29T10:57:55.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.330+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7f0038054420 con 0x7f003c06d8c0 2024-03-29T10:57:55.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.330+0000 7f0031ffb640 10 client.4519.objecter ms_dispatch 0x7f003c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:57:55.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.330+0000 7f0031ffb640 3 client.4519.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:57:55.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.330+0000 7f0031ffb640 3 client.4519.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:57:55.311 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.331+0000 7f0031ffb640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T10:57:55.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.331+0000 7f0040be5640 10 monclient: get_auth_request con 0x7f002006dff0 auth_method 0 2024-03-29T10:57:55.312 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.332+0000 7f0031ffb640 10 client.4519.objecter ms_handle_connect 0x7f002006dff0 2024-03-29T10:57:55.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.336+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00380506c0 con 0x7f003c06d8c0 2024-03-29T10:57:55.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.336+0000 7f0031ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:55.317 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.336+0000 7f0031ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:55.654 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:57:55.654 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:55.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.670+0000 7f004364a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:55.654 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.670+0000 7f004364a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:55.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.670+0000 7f004364a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:57:55.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.670+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f003c074e10 con 0x7f003c06d8c0 2024-03-29T10:57:55.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.672+0000 7f0031ffb640 1 -- 172.21.15.79:0/4188071482 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7f00380506c0 con 0x7f003c06d8c0 2024-03-29T10:57:55.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.672+0000 7f0031ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:55.655 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.672+0000 7f0031ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:55.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.675+0000 7f004364a640 10 librados: watch_flush enter 2024-03-29T10:57:55.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.675+0000 7f004364a640 10 librados: watch_flush exit 2024-03-29T10:57:55.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.675+0000 7f004364a640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T10:57:55.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.675+0000 7f004364a640 10 client.4519.objecter successfully canceled tick 2024-03-29T10:57:55.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 >> v1:172.21.15.79:6816/4290694627 conn(0x7f002006dff0 legacy=0x7f00200704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.656 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 10 monclient: shutdown 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 >> v1:172.21.15.82:6789/0 conn(0x7f003c06d8c0 legacy=0x7f003c135af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 shutdown_connections 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 >> 172.21.15.79:0/4188071482 conn(0x7f003c0b3a00 msgr2=0x7f003c14d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 shutdown_connections 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 -- 172.21.15.79:0/4188071482 wait complete. 2024-03-29T10:57:55.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:57:55.676+0000 7f004364a640 1 librados: shutdown 2024-03-29T10:57:55.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:02.481 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:02.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.642+0000 7fcf66d64640 10 monclient: get_monmap_and_config 2024-03-29T10:58:02.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.642+0000 7fcf66d64640 10 monclient: build_initial_monmap 2024-03-29T10:58:02.623 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.642+0000 7fcf66d64640 10 monclient: monmap: 2024-03-29T10:58:02.623 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:02.623 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:02.643864+0000 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:02.643864+0000 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:02.624 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.645+0000 7fcf66d64640 1 Processor -- start 2024-03-29T10:58:02.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.645+0000 7fcf66d64640 1 -- start start 2024-03-29T10:58:02.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.645+0000 7fcf66d64640 10 monclient: init 2024-03-29T10:58:02.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:02.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:02.626 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient(hunting): picked mon.noname-a con 0x7fcf6006d9d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient(hunting): picked mon.noname-b con 0x7fcf6013ef30 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient(hunting): picked mon.noname-c con 0x7fcf600b07d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf600aed10 con 0x7fcf6006d9d0 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf600b0c40 con 0x7fcf600b07d0 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf600ab1b0 con 0x7fcf6013ef30 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:02.627 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.646+0000 7fcf66d64640 10 monclient(hunting): authenticate will time out at 1601.938832s 2024-03-29T10:58:02.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.649+0000 7fcf5ffff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fcf6013ef30 0x7fcf600af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:41786/0 (socket says 172.21.15.79:41786) 2024-03-29T10:58:02.629 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.649+0000 7fcf5ffff640 1 -- 172.21.15.79:0/2422480068 learned_addr learned my addr 172.21.15.79:0/2422480068 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:02.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908869729 0 0) 0x7fcf600ab1b0 con 0x7fcf6013ef30 2024-03-29T10:58:02.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:02.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:02.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:02.630 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): my global_id is 4679 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf44003520 con 0x7fcf6013ef30 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550125303 0 0) 0x7fcf600aed10 con 0x7fcf6006d9d0 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf600ab1b0 con 0x7fcf6006d9d0 2024-03-29T10:58:02.631 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116554245 0 0) 0x7fcf44003520 con 0x7fcf6013ef30 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 >> v1:172.21.15.82:6789/0 conn(0x7fcf600b07d0 legacy=0x7fcf6014bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 >> v1:172.21.15.79:6789/0 conn(0x7fcf6006d9d0 legacy=0x7fcf6013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf600ab490 con 0x7fcf6013ef30 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.650+0000 7fcf5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf66d64640 5 monclient: authenticate success, global_id 4679 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf66d64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcf50001690 con 0x7fcf6013ef30 2024-03-29T10:58:02.632 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: dump: 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf66d64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf50001860 con 0x7fcf6013ef30 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:02.633 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf66d64640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf66d64640 4 set_mon_vals no callback set 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 1 -- 172.21.15.79:0/2422480068 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcf50001bf0 con 0x7fcf6013ef30 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.651+0000 7fcf5f7fe640 10 monclient: dump: 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:02.634 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.635 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.652+0000 7fcf66d64640 10 monclient: shutdown 2024-03-29T10:58:02.635 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.652+0000 7fcf66d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:02.635 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.652+0000 7fcf66d64640 1 -- 172.21.15.79:0/2422480068 >> v1:172.21.15.79:6790/0 conn(0x7fcf6013ef30 legacy=0x7fcf600af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:02.635 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.652+0000 7fcf66d64640 1 -- 172.21.15.79:0/2422480068 shutdown_connections 2024-03-29T10:58:02.635 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.652+0000 7fcf66d64640 1 -- 172.21.15.79:0/2422480068 >> 172.21.15.79:0/2422480068 conn(0x7fcf6006b660 msgr2=0x7fcf6006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:02.635 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.654+0000 7fcf66d64640 1 -- 172.21.15.79:0/2422480068 shutdown_connections 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.654+0000 7fcf66d64640 1 -- 172.21.15.79:0/2422480068 wait complete. 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.655+0000 7fcf66d64640 10 monclient: build_initial_monmap 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.655+0000 7fcf66d64640 10 monclient: monmap: 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:02.636 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.655+0000 7fcf66d64640 1 librados: starting msgr at 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.655+0000 7fcf66d64640 1 librados: starting objecter 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.655+0000 7fcf66d64640 1 Processor -- start 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 1 -- start start 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 1 librados: setting wanted keys 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 1 librados: calling monclient init 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 10 monclient: init 2024-03-29T10:58:02.638 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.656+0000 7fcf66d64640 10 monclient(hunting): picked mon.noname-a con 0x7fcf6006d9d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 10 monclient(hunting): picked mon.noname-b con 0x7fcf600b07d0 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 10 monclient(hunting): picked mon.noname-c con 0x7fcf6013ef30 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf601d53d0 con 0x7fcf6006d9d0 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf600b4010 con 0x7fcf600b07d0 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf601365d0 con 0x7fcf6013ef30 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:02.639 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.657+0000 7fcf66d64640 10 monclient(hunting): authenticate will time out at 1601.949151s 2024-03-29T10:58:02.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.660+0000 7fcf65301640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fcf6013ef30 0x7fcf60130640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:44576/0 (socket says 172.21.15.79:44576) 2024-03-29T10:58:02.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.660+0000 7fcf65301640 1 -- 172.21.15.79:0/2057517343 learned_addr learned my addr 172.21.15.79:0/2057517343 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:02.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.660+0000 7fcf5d7fa640 10 client.?.objecter ms_handle_connect 0x7fcf6006d9d0 2024-03-29T10:58:02.641 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.660+0000 7fcf5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 client.?.objecter ms_handle_connect 0x7fcf6013ef30 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807544911 0 0) 0x7fcf601d53d0 con 0x7fcf6006d9d0 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:02.646 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf48004eb0 con 0x7fcf6006d9d0 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533614465 0 0) 0x7fcf601365d0 con 0x7fcf6013ef30 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 10 monclient(hunting): my global_id is 4682 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.661+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf48006230 con 0x7fcf6013ef30 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1493682698 0 0) 0x7fcf48006230 con 0x7fcf6013ef30 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf601365d0 con 0x7fcf6013ef30 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcf58001cf0 con 0x7fcf6013ef30 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): dump: 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:02.647 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952409 0 0) 0x7fcf48004eb0 con 0x7fcf6006d9d0 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf48006230 con 0x7fcf6006d9d0 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcf4c0017a0 con 0x7fcf6006d9d0 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 10 monclient(hunting): dump: 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:02.648 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 408192900 0 0) 0x7fcf601365d0 con 0x7fcf6013ef30 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.662+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 >> v1:172.21.15.82:6789/0 conn(0x7fcf600b07d0 legacy=0x7fcf601d5860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 >> v1:172.21.15.79:6789/0 conn(0x7fcf6006d9d0 legacy=0x7fcf600b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 1 monclient: found mon.c 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:02.649 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf600b2dd0 con 0x7fcf6013ef30 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 5 monclient: authenticate success, global_id 4682 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 10 monclient: _renew_subs 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf600b2fe0 con 0x7fcf6013ef30 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 10 client.4682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 10 monclient: _renew_subs 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf600b1bf0 con 0x7fcf6013ef30 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.663+0000 7fcf66d64640 1 librados: init done 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.664+0000 7fcf66d64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf66d64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf66d64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:02.650 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf60075f10 con 0x7fcf6013ef30 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf58003990 con 0x7fcf6013ef30 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fcf58003ce0 con 0x7fcf6013ef30 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.665+0000 7fcf5d7fa640 10 monclient: dump: 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:02.651 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:02.652 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.666+0000 7fcf5e7fc640 4 set_mon_vals no callback set 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.667+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fcf58003ec0 con 0x7fcf6013ef30 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.668+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7fcf58084df0 con 0x7fcf6013ef30 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.668+0000 7fcf5d7fa640 10 client.4682.objecter ms_dispatch 0x7fcf60000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.668+0000 7fcf5d7fa640 3 client.4682.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.668+0000 7fcf5d7fa640 3 client.4682.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.668+0000 7fcf5d7fa640 20 client.4682.objecter dump_active .. 0 homeless 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.668+0000 7fcf64b00640 10 monclient: get_auth_request con 0x7fcf48072610 auth_method 0 2024-03-29T10:58:02.653 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.669+0000 7fcf5d7fa640 10 client.4682.objecter ms_handle_connect 0x7fcf48072610 2024-03-29T10:58:02.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.677+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf580506d0 con 0x7fcf6013ef30 2024-03-29T10:58:02.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.677+0000 7fcf5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:02.657 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:02.677+0000 7fcf5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:02.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.004+0000 7fcf66d64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:02.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.004+0000 7fcf66d64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:02.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.004+0000 7fcf66d64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:02.985 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.004+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf601d5560 con 0x7fcf6013ef30 2024-03-29T10:58:02.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.006+0000 7fcf5d7fa640 1 -- 172.21.15.79:0/2057517343 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7fcf58052600 con 0x7fcf6013ef30 2024-03-29T10:58:02.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.006+0000 7fcf5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:02.986 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.006+0000 7fcf5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:02.986 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:02.987 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 10 librados: watch_flush enter 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 10 librados: watch_flush exit 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 20 client.4682.objecter shutdown clearing up homeless session... 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 10 client.4682.objecter successfully canceled tick 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 >> v1:172.21.15.79:6816/4290694627 conn(0x7fcf48072610 legacy=0x7fcf48074ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 10 monclient: shutdown 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:02.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.009+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 >> v1:172.21.15.79:6790/0 conn(0x7fcf6013ef30 legacy=0x7fcf60130640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:02.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.010+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 shutdown_connections 2024-03-29T10:58:02.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.010+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 >> 172.21.15.79:0/2057517343 conn(0x7fcf6006b660 msgr2=0x7fcf6013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:02.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.010+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 shutdown_connections 2024-03-29T10:58:02.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.010+0000 7fcf66d64640 1 -- 172.21.15.79:0/2057517343 wait complete. 2024-03-29T10:58:02.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:03.010+0000 7fcf66d64640 1 librados: shutdown 2024-03-29T10:58:03.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:09.814 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.982+0000 7fe7dafaf640 10 monclient: get_monmap_and_config 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.982+0000 7fe7dafaf640 10 monclient: build_initial_monmap 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.982+0000 7fe7dafaf640 10 monclient: monmap: 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:09.983496+0000 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:09.983496+0000 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:09.963 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:09.964 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:09.964 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:09.964 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:09.964 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:09.965 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.985+0000 7fe7dafaf640 1 Processor -- start 2024-03-29T10:58:09.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.985+0000 7fe7dafaf640 1 -- start start 2024-03-29T10:58:09.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.985+0000 7fe7dafaf640 10 monclient: init 2024-03-29T10:58:09.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:09.966 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d41422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d4142690 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d40b17d0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d4142c80 con 0x7fe7d40b17d0 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40aae40 con 0x7fe7d41422b0 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d413fbd0 con 0x7fe7d4142690 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:09.967 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.986+0000 7fe7dafaf640 10 monclient(hunting): authenticate will time out at 1609.278819s 2024-03-29T10:58:09.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.990+0000 7fe7d3fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7fe7d4142690 0x7fe7d40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:43468/0 (socket says 172.21.15.79:43468) 2024-03-29T10:58:09.970 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.990+0000 7fe7d3fff640 1 -- 172.21.15.79:0/1304110223 learned_addr learned my addr 172.21.15.79:0/1304110223 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:09.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365455009 0 0) 0x7fe7d40aae40 con 0x7fe7d41422b0 2024-03-29T10:58:09.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:09.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:09.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:09.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:58:09.973 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7bc003520 con 0x7fe7d41422b0 2024-03-29T10:58:09.974 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068303987 0 0) 0x7fe7d4142c80 con 0x7fe7d40b17d0 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d40aae40 con 0x7fe7d40b17d0 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751067108 0 0) 0x7fe7bc003520 con 0x7fe7d41422b0 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 >> v1:172.21.15.79:6790/0 conn(0x7fe7d4142690 legacy=0x7fe7d40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 >> v1:172.21.15.79:6789/0 conn(0x7fe7d40b17d0 legacy=0x7fe7d40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 monclient: found mon.noname-c 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d4151940 con 0x7fe7d41422b0 2024-03-29T10:58:09.975 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7dafaf640 5 monclient: authenticate success, global_id 4525 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7dafaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe7c8001690 con 0x7fe7d41422b0 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7d37fe640 10 monclient: dump: 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:09.976 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.993+0000 7fe7dafaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7c80018b0 con 0x7fe7d41422b0 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7dafaf640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7dafaf640 4 set_mon_vals no callback set 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7d37fe640 1 -- 172.21.15.79:0/1304110223 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe7c8001bc0 con 0x7fe7d41422b0 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.994+0000 7fe7d37fe640 10 monclient: dump: 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:09.977 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:09.978 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:09.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.995+0000 7fe7dafaf640 10 monclient: shutdown 2024-03-29T10:58:09.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.995+0000 7fe7dafaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:09.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.995+0000 7fe7dafaf640 1 -- 172.21.15.79:0/1304110223 >> v1:172.21.15.82:6789/0 conn(0x7fe7d41422b0 legacy=0x7fe7d413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:09.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.995+0000 7fe7dafaf640 1 -- 172.21.15.79:0/1304110223 shutdown_connections 2024-03-29T10:58:09.979 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.995+0000 7fe7dafaf640 1 -- 172.21.15.79:0/1304110223 >> 172.21.15.79:0/1304110223 conn(0x7fe7d4066a00 msgr2=0x7fe7d4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:09.982 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.997+0000 7fe7dafaf640 1 -- 172.21.15.79:0/1304110223 shutdown_connections 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.997+0000 7fe7dafaf640 1 -- 172.21.15.79:0/1304110223 wait complete. 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 10 monclient: build_initial_monmap 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 10 monclient: monmap: 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 1 librados: starting msgr at 2024-03-29T10:58:09.983 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 1 librados: starting objecter 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 1 Processor -- start 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 1 -- start start 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 1 librados: setting wanted keys 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 1 librados: calling monclient init 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:09.998+0000 7fe7dafaf640 10 monclient: init 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe7d40b17d0 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe7d41422b0 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe7d4142690 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d4071490 con 0x7fe7d4142690 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40ab6a0 con 0x7fe7d41422b0 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d40ab980 con 0x7fe7d40b17d0 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:09.984 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.000+0000 7fe7dafaf640 10 monclient(hunting): authenticate will time out at 1609.292250s 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.003+0000 7fe7d3fff640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7fe7d41422b0 0x7fe7d406e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:40257/0 (socket says 172.21.15.79:40257) 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.003+0000 7fe7d3fff640 1 -- 172.21.15.79:0/3161410140 learned_addr learned my addr 172.21.15.79:0/3161410140 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 client.?.objecter ms_handle_connect 0x7fe7d41422b0 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 client.?.objecter ms_handle_connect 0x7fe7d40b17d0 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979865721 0 0) 0x7fe7d40ab6a0 con 0x7fe7d41422b0 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:09.989 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.007+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c0004eb0 con 0x7fe7d41422b0 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 client.?.objecter ms_handle_connect 0x7fe7d4142690 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031217090 0 0) 0x7fe7c0004eb0 con 0x7fe7d41422b0 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c0005300 con 0x7fe7d41422b0 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe7c40017c0 con 0x7fe7d41422b0 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): dump: 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.990 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162920989 0 0) 0x7fe7d40ab980 con 0x7fe7d40b17d0 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:09.991 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 10 monclient(hunting): my global_id is 4688 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.008+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c0005880 con 0x7fe7d40b17d0 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 608094886 0 0) 0x7fe7c0005300 con 0x7fe7d41422b0 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 >> v1:172.21.15.79:6790/0 conn(0x7fe7d40b17d0 legacy=0x7fe7d4070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 >> v1:172.21.15.79:6789/0 conn(0x7fe7d4142690 legacy=0x7fe7d406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 1 monclient: found mon.b 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d406fea0 con 0x7fe7d41422b0 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 10 monclient: _finish_auth 0 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 5 monclient: authenticate success, global_id 4528 2024-03-29T10:58:09.992 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 10 monclient: _renew_subs 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d4071090 con 0x7fe7d41422b0 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 10 monclient: _renew_subs 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d4200c30 con 0x7fe7d41422b0 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7dafaf640 1 librados: init done 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7c4001cb0 con 0x7fe7d41422b0 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.009+0000 7fe7d2ffd640 4 set_mon_vals no callback set 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.010+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7fe7c40017c0 con 0x7fe7d41422b0 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.010+0000 7fe7d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.010+0000 7fe7d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.010+0000 7fe7d17fa640 10 monclient: dump: 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:09.993 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:09.994 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7fe7c40048a0 con 0x7fe7d41422b0 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7fe7c4054500 con 0x7fe7d41422b0 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d17fa640 10 client.4528.objecter ms_dispatch 0x7fe7d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d17fa640 3 client.4528.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d17fa640 3 client.4528.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d17fa640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.012+0000 7fe7d8d4b640 10 monclient: get_auth_request con 0x7fe7c006e490 auth_method 0 2024-03-29T10:58:09.995 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.013+0000 7fe7d17fa640 10 client.4528.objecter ms_handle_connect 0x7fe7c006e490 2024-03-29T10:58:09.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.013+0000 7fe7dafaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:09.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.013+0000 7fe7dafaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:09.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.013+0000 7fe7dafaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:58:09.996 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.013+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4005080 con 0x7fe7d41422b0 2024-03-29T10:58:10.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.019+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7c40507a0 con 0x7fe7d41422b0 2024-03-29T10:58:10.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.019+0000 7fe7d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:10.000 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.019+0000 7fe7d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:10.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.363+0000 7fe7dafaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.364+0000 7fe7dafaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.364+0000 7fe7dafaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.82:6789/0 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.364+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7b4005330 con 0x7fe7d41422b0 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.364+0000 7fe7d17fa640 1 -- 172.21.15.79:0/3161410140 <== mon.1 v1:172.21.15.82:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7fe7c4050940 con 0x7fe7d41422b0 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.364+0000 7fe7d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.364+0000 7fe7d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:10.345 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:10.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.368+0000 7fe7dafaf640 10 librados: watch_flush enter 2024-03-29T10:58:10.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.368+0000 7fe7dafaf640 10 librados: watch_flush exit 2024-03-29T10:58:10.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.368+0000 7fe7dafaf640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T10:58:10.349 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.368+0000 7fe7dafaf640 10 client.4528.objecter successfully canceled tick 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.372+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 >> v1:172.21.15.79:6816/4290694627 conn(0x7fe7c006e490 legacy=0x7fe7c0070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.372+0000 7fe7dafaf640 10 monclient: shutdown 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.372+0000 7fe7dafaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.372+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 >> v1:172.21.15.82:6789/0 conn(0x7fe7d41422b0 legacy=0x7fe7d406e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.373+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 shutdown_connections 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.373+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 >> 172.21.15.79:0/3161410140 conn(0x7fe7d4066a00 msgr2=0x7fe7d413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:10.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.373+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 shutdown_connections 2024-03-29T10:58:10.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.373+0000 7fe7dafaf640 1 -- 172.21.15.79:0/3161410140 wait complete. 2024-03-29T10:58:10.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:10.374+0000 7fe7dafaf640 1 librados: shutdown 2024-03-29T10:58:10.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:17.186 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.356+0000 7f829df8f640 10 monclient: get_monmap_and_config 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.356+0000 7f829df8f640 10 monclient: build_initial_monmap 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.356+0000 7f829df8f640 10 monclient: monmap: 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:17.358403+0000 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:17.358403+0000 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:17.338 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.339 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:17.339 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:17.339 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:17.339 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.339 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.357+0000 7f829df8f640 1 Processor -- start 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.357+0000 7f829df8f640 1 -- start start 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.357+0000 7f829df8f640 10 monclient: init 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient(hunting): picked mon.noname-c con 0x7f82980b0500 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient(hunting): picked mon.noname-a con 0x7f82980b08e0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient(hunting): picked mon.noname-b con 0x7f82980aeca0 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8298140620 con 0x7f82980b08e0 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829813daf0 con 0x7f82980b0500 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980ae180 con 0x7f82980aeca0 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f829df8f640 10 monclient(hunting): authenticate will time out at 1616.651627s 2024-03-29T10:58:17.340 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f8297fff640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f82980aeca0 0x7f829814b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:56490/0 (socket says 172.21.15.79:56490) 2024-03-29T10:58:17.341 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.358+0000 7f8297fff640 1 -- 172.21.15.79:0/402362909 learned_addr learned my addr 172.21.15.79:0/402362909 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:17.342 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299316986 0 0) 0x7f829813daf0 con 0x7f82980b0500 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8280003520 con 0x7f82980b0500 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934860289 0 0) 0x7f8298140620 con 0x7f82980b08e0 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8280003840 con 0x7f82980b08e0 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.359+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556848723 0 0) 0x7f8280003520 con 0x7f82980b0500 2024-03-29T10:58:17.343 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 >> v1:172.21.15.79:6790/0 conn(0x7f82980aeca0 legacy=0x7f829814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 >> v1:172.21.15.79:6789/0 conn(0x7f82980b08e0 legacy=0x7f829813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 monclient: found mon.noname-c 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980accf0 con 0x7f82980b0500 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:17.344 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8288001670 con 0x7f82980b0500 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: dump: 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:17.345 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f829df8f640 5 monclient: authenticate success, global_id 4531 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f829df8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8288001850 con 0x7f82980b0500 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f829df8f640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f829df8f640 4 set_mon_vals no callback set 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 1 -- 172.21.15.79:0/402362909 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8288001670 con 0x7f82980b0500 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.360+0000 7f82967fc640 10 monclient: dump: 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.346 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 10 monclient: shutdown 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 1 -- 172.21.15.79:0/402362909 >> v1:172.21.15.82:6789/0 conn(0x7f82980b0500 legacy=0x7f829813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 1 -- 172.21.15.79:0/402362909 shutdown_connections 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 1 -- 172.21.15.79:0/402362909 >> 172.21.15.79:0/402362909 conn(0x7f829806b660 msgr2=0x7f829806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:17.347 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 1 -- 172.21.15.79:0/402362909 shutdown_connections 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.361+0000 7f829df8f640 1 -- 172.21.15.79:0/402362909 wait complete. 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.362+0000 7f829df8f640 10 monclient: build_initial_monmap 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.362+0000 7f829df8f640 10 monclient: monmap: 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:17.352 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 1 librados: starting msgr at 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 1 librados: starting objecter 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 1 Processor -- start 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 1 -- start start 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 1 librados: setting wanted keys 2024-03-29T10:58:17.353 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 1 librados: calling monclient init 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.363+0000 7f829df8f640 10 monclient: init 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient(hunting): picked mon.noname-a con 0x7f82980aeca0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient(hunting): picked mon.noname-c con 0x7f829813abd0 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f82977fe640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f82980aeca0 0x7f82981d3500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:51150/0 (socket says 172.21.15.79:51150) 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8298070590 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f82977fe640 1 -- 172.21.15.79:0/2014817729 learned_addr learned my addr 172.21.15.79:0/2014817729 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82981d70b0 con 0x7f82980aeca0 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82981d3c10 con 0x7f8298070590 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829813a920 con 0x7f829813abd0 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:17.354 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f829df8f640 10 monclient(hunting): authenticate will time out at 1616.657866s 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f827ffff640 10 client.?.objecter ms_handle_connect 0x7f82980aeca0 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f827ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f827ffff640 10 client.?.objecter ms_handle_connect 0x7f829813abd0 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.364+0000 7f827ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 client.?.objecter ms_handle_connect 0x7f8298070590 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318054109 0 0) 0x7f829813a920 con 0x7f829813abd0 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:17.355 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): my global_id is 4691 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8284004eb0 con 0x7f829813abd0 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318320096 0 0) 0x7f82981d70b0 con 0x7f82980aeca0 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8284006230 con 0x7f82980aeca0 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014435172 0 0) 0x7f82981d3c10 con 0x7f8298070590 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.365+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82981d70b0 con 0x7f8298070590 2024-03-29T10:58:17.356 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809558010 0 0) 0x7f8284004eb0 con 0x7f829813abd0 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82981d3c10 con 0x7f829813abd0 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f828c001890 con 0x7f829813abd0 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): dump: 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.357 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 877722517 0 0) 0x7f8284006230 con 0x7f82980aeca0 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8284004eb0 con 0x7f82980aeca0 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8288001670 con 0x7f82980aeca0 2024-03-29T10:58:17.358 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): dump: 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.359 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2134009153 0 0) 0x7f82981d70b0 con 0x7f8298070590 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8284006230 con 0x7f8298070590 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f8290001a60 con 0x7f8298070590 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient(hunting): dump: 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.360 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3811874184 0 0) 0x7f82981d3c10 con 0x7f829813abd0 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 >> v1:172.21.15.82:6789/0 conn(0x7f8298070590 legacy=0x7f829813a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 >> v1:172.21.15.79:6789/0 conn(0x7f82980aeca0 legacy=0x7f82981d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:17.361 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 monclient: found mon.c 2024-03-29T10:58:17.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:17.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:17.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82981d6a00 con 0x7f829813abd0 2024-03-29T10:58:17.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:17.362 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:17.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f827ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:17.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 5 monclient: authenticate success, global_id 4691 2024-03-29T10:58:17.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 10 monclient: _renew_subs 2024-03-29T10:58:17.363 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:17.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82981d6b90 con 0x7f829813abd0 2024-03-29T10:58:17.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:17.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 10 monclient: _renew_subs 2024-03-29T10:58:17.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:17.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82981d7720 con 0x7f829813abd0 2024-03-29T10:58:17.364 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.366+0000 7f829df8f640 1 librados: init done 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f829df8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f828c0035f0 con 0x7f829813abd0 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f827ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f828c003900 con 0x7f829813abd0 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f827ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f827ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f827ffff640 10 monclient: dump: 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:17.365 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.368+0000 7f8295ffb640 4 set_mon_vals no callback set 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.369+0000 7f829df8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.369+0000 7f829df8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.369+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82980ad0c0 con 0x7f829813abd0 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.369+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f828c0198d0 con 0x7f829813abd0 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f82977fe640 10 monclient: get_auth_request con 0x7f828406e340 auth_method 0 2024-03-29T10:58:17.366 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f827ffff640 10 client.4691.objecter ms_handle_connect 0x7f828406e340 2024-03-29T10:58:17.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7f828c0534f0 con 0x7f829813abd0 2024-03-29T10:58:17.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f827ffff640 10 client.4691.objecter ms_dispatch 0x7f8298000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:17.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f827ffff640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:17.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f827ffff640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:17.367 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.370+0000 7f827ffff640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-29T10:58:17.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.379+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f828c04f790 con 0x7f829813abd0 2024-03-29T10:58:17.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.379+0000 7f827ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:17.368 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.379+0000 7f827ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.748+0000 7f829df8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.748+0000 7f829df8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.748+0000 7f829df8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.748+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f829811f240 con 0x7f829813abd0 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.749+0000 7f827ffff640 1 -- 172.21.15.79:0/2014817729 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7f828c04f790 con 0x7f829813abd0 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.749+0000 7f827ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:17.730 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.749+0000 7f827ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:17.731 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:17.731 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 10 librados: watch_flush enter 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 10 librados: watch_flush exit 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 10 client.4691.objecter successfully canceled tick 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 >> v1:172.21.15.79:6816/4290694627 conn(0x7f828406e340 legacy=0x7f8284070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 10 monclient: shutdown 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:17.734 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.752+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 >> v1:172.21.15.79:6790/0 conn(0x7f829813abd0 legacy=0x7f8298139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:17.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.753+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 shutdown_connections 2024-03-29T10:58:17.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.753+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 >> 172.21.15.79:0/2014817729 conn(0x7f829806b660 msgr2=0x7f82980b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:17.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.753+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 shutdown_connections 2024-03-29T10:58:17.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.753+0000 7f829df8f640 1 -- 172.21.15.79:0/2014817729 wait complete. 2024-03-29T10:58:17.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:17.753+0000 7f829df8f640 1 librados: shutdown 2024-03-29T10:58:17.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:24.558 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.730+0000 7f1ce7fff640 10 monclient: get_monmap_and_config 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.730+0000 7f1ce7fff640 10 monclient: build_initial_monmap 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.730+0000 7f1ce7fff640 10 monclient: monmap: 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:24.731064+0000 2024-03-29T10:58:24.713 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:24.731064+0000 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:24.714 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.733+0000 7f1ce7fff640 1 Processor -- start 2024-03-29T10:58:24.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.733+0000 7f1ce7fff640 1 -- start start 2024-03-29T10:58:24.715 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.733+0000 7f1ce7fff640 10 monclient: init 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce0006e30 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce0035e00 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce00c9670 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce0035970 con 0x7f1ce0006e30 2024-03-29T10:58:24.718 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce0036280 con 0x7f1ce00c9670 2024-03-29T10:58:24.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce0035640 con 0x7f1ce0035e00 2024-03-29T10:58:24.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:24.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.734+0000 7f1ce7fff640 10 monclient(hunting): authenticate will time out at 1624.026970s 2024-03-29T10:58:24.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.738+0000 7f1ce6ffd640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f1ce0006e30 0x7f1ce00cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:34471/0 (socket says 172.21.15.79:34471) 2024-03-29T10:58:24.719 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.738+0000 7f1ce6ffd640 1 -- 172.21.15.79:0/3324005814 learned_addr learned my addr 172.21.15.79:0/3324005814 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:24.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.741+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209288913 0 0) 0x7f1ce0035970 con 0x7f1ce0006e30 2024-03-29T10:58:24.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.741+0000 7f1ce5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:24.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.741+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:24.735 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.741+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.741+0000 7f1ce5ffb640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.741+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd0003520 con 0x7f1ce0006e30 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3096739180 0 0) 0x7f1cd0003520 con 0x7f1ce0006e30 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 >> v1:172.21.15.79:6790/0 conn(0x7f1ce0035e00 legacy=0x7f1ce00c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 >> v1:172.21.15.82:6789/0 conn(0x7f1ce00c9670 legacy=0x7f1ce00d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 monclient: found mon.noname-a 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.79:6789/0 2024-03-29T10:58:24.736 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce0034e20 con 0x7f1ce0006e30 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1ce804f730 con 0x7f1ce0006e30 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: dump: 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.737 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 <== mon.0 v1:172.21.15.79:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce80008d0 con 0x7f1ce0006e30 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 1 -- 172.21.15.79:0/3324005814 <== mon.0 v1:172.21.15.79:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1ce80758b0 con 0x7f1ce0006e30 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce5ffb640 10 monclient: dump: 2024-03-29T10:58:24.738 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce7fff640 5 monclient: authenticate success, global_id 4599 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce7fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:24.739 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.742+0000 7f1ce7fff640 4 set_mon_vals no callback set 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.743+0000 7f1ce7fff640 10 monclient: shutdown 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.743+0000 7f1ce7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.743+0000 7f1ce7fff640 1 -- 172.21.15.79:0/3324005814 >> v1:172.21.15.79:6789/0 conn(0x7f1ce0006e30 legacy=0x7f1ce00cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.743+0000 7f1ce7fff640 1 -- 172.21.15.79:0/3324005814 shutdown_connections 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.743+0000 7f1ce7fff640 1 -- 172.21.15.79:0/3324005814 >> 172.21.15.79:0/3324005814 conn(0x7f1ce0005c00 msgr2=0x7f1ce0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.744+0000 7f1ce7fff640 1 -- 172.21.15.79:0/3324005814 shutdown_connections 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.747+0000 7f1ce7fff640 1 -- 172.21.15.79:0/3324005814 wait complete. 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.748+0000 7f1ce7fff640 10 monclient: build_initial_monmap 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.748+0000 7f1ce7fff640 10 monclient: monmap: 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:24.740 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 1 librados: starting msgr at 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 1 librados: starting objecter 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 1 Processor -- start 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 1 -- start start 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 1 librados: setting wanted keys 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 1 librados: calling monclient init 2024-03-29T10:58:24.741 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.749+0000 7f1ce7fff640 10 monclient: init 2024-03-29T10:58:24.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.750+0000 7f1ce7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:24.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.750+0000 7f1ce7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:24.742 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.750+0000 7f1ce7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce0006e30 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:24.748 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.755+0000 7f1ce7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce0035e00 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce6ffd640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f1ce0006e30 0x7f1ce00cb240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:39925/0 (socket says 172.21.15.79:39925) 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce00c9670 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce7fff640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00cd890 con 0x7f1ce00c9670 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce7fff640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00cdb70 con 0x7f1ce0006e30 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce7fff640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce0033c20 con 0x7f1ce0035e00 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce7fff640 10 monclient(hunting): authenticate will time out at 1624.048843s 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.756+0000 7f1ce6ffd640 1 -- 172.21.15.79:0/1142465081 learned_addr learned my addr 172.21.15.79:0/1142465081 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.757+0000 7f1ccf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ce0006e30 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.757+0000 7f1ccf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280750319 0 0) 0x7f1ce00cdb70 con 0x7f1ce0006e30 2024-03-29T10:58:24.749 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4004eb0 con 0x7f1ce0006e30 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ce0035e00 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ce00c9670 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106202569 0 0) 0x7f1ce00cd890 con 0x7f1ce00c9670 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:24.750 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 10 monclient(hunting): my global_id is 4602 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.758+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd40051d0 con 0x7f1ce00c9670 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1070056676 0 0) 0x7f1cd40051d0 con 0x7f1ce00c9670 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce00cd890 con 0x7f1ce00c9670 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1cdc001810 con 0x7f1ce00c9670 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient(hunting): dump: 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.751 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3068374697 0 0) 0x7f1ce00cd890 con 0x7f1ce00c9670 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 >> v1:172.21.15.79:6790/0 conn(0x7f1ce0035e00 legacy=0x7f1ce0033510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 >> v1:172.21.15.82:6789/0 conn(0x7f1ce0006e30 legacy=0x7f1ce00cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:24.752 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 monclient: found mon.a 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce0033db0 con 0x7f1ce00c9670 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 5 monclient: authenticate success, global_id 4602 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 10 monclient: _renew_subs 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce00ce270 con 0x7f1ce00c9670 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 10 monclient: _renew_subs 2024-03-29T10:58:24.753 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce0032c70 con 0x7f1ce00c9670 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.759+0000 7f1ce7fff640 1 librados: init done 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.762+0000 7f1ccd7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.762+0000 7f1ccd7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.762+0000 7f1ccd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.762+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0007b70 con 0x7f1ce00c9670 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cdc003610 con 0x7f1ce00c9670 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f1cdc001d80 con 0x7f1ce00c9670 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:24.754 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 10 monclient: dump: 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f1cdc003a60 con 0x7f1ce00c9670 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7f1cdc0838e0 con 0x7f1ce00c9670 2024-03-29T10:58:24.755 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 10 client.4602.objecter ms_dispatch 0x7f1ce8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:24.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:24.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:24.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ccf7fe640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T10:58:24.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.763+0000 7f1ce57fa640 4 set_mon_vals no callback set 2024-03-29T10:58:24.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.764+0000 7f1ce6ffd640 10 monclient: get_auth_request con 0x7f1cd406e010 auth_method 0 2024-03-29T10:58:24.756 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.765+0000 7f1ccf7fe640 10 client.4602.objecter ms_handle_connect 0x7f1cd406e010 2024-03-29T10:58:24.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.772+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cdc04f240 con 0x7f1ce00c9670 2024-03-29T10:58:24.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.772+0000 7f1ccf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:24.757 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:24.772+0000 7f1ccf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:25.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.126+0000 7f1ccd7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:25.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.126+0000 7f1ccd7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:25.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.126+0000 7f1ccd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:25.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.126+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce0005120 con 0x7f1ce00c9670 2024-03-29T10:58:25.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.127+0000 7f1ccf7fe640 1 -- 172.21.15.79:0/1142465081 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7f1cdc051170 con 0x7f1ce00c9670 2024-03-29T10:58:25.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.127+0000 7f1ccf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:25.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.127+0000 7f1ccf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:25.127 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:25.127 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:25.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 10 librados: watch_flush enter 2024-03-29T10:58:25.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 10 librados: watch_flush exit 2024-03-29T10:58:25.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T10:58:25.128 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 10 client.4602.objecter successfully canceled tick 2024-03-29T10:58:25.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 >> v1:172.21.15.79:6816/4290694627 conn(0x7f1cd406e010 legacy=0x7f1cd40704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:25.129 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 10 monclient: shutdown 2024-03-29T10:58:25.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:25.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 >> v1:172.21.15.79:6789/0 conn(0x7f1ce00c9670 legacy=0x7f1ce016b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:25.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 shutdown_connections 2024-03-29T10:58:25.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 >> 172.21.15.79:0/1142465081 conn(0x7f1ce0005c00 msgr2=0x7f1ce00c4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:25.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 shutdown_connections 2024-03-29T10:58:25.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 -- 172.21.15.79:0/1142465081 wait complete. 2024-03-29T10:58:25.131 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:25.131+0000 7f1ccd7fa640 1 librados: shutdown 2024-03-29T10:58:25.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:31.939 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:32.089 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.109+0000 7f5a382b3640 10 monclient: get_monmap_and_config 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.109+0000 7f5a382b3640 10 monclient: build_initial_monmap 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.109+0000 7f5a382b3640 10 monclient: monmap: 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:32.110067+0000 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:32.110067+0000 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:32.090 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:32.091 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:32.091 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.110+0000 7f5a382b3640 1 Processor -- start 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.112+0000 7f5a382b3640 1 -- start start 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.112+0000 7f5a382b3640 10 monclient: init 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.113+0000 7f5a382b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.113+0000 7f5a382b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5a300708a0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 10 monclient(hunting): picked mon.noname-c con 0x7f5a3006fe90 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5a30143c50 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a30072630 con 0x7f5a300708a0 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a30071410 con 0x7f5a3006fe90 2024-03-29T10:58:32.095 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300afaa0 con 0x7f5a30143c50 2024-03-29T10:58:32.096 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:32.096 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.114+0000 7f5a382b3640 10 monclient(hunting): authenticate will time out at 1631.406221s 2024-03-29T10:58:32.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.117+0000 7f5a3584e640 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f5a3006fe90 0x7f5a30143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.79:36757/0 (socket says 172.21.15.79:36757) 2024-03-29T10:58:32.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.117+0000 7f5a3584e640 1 -- 172.21.15.79:0/1267909291 learned_addr learned my addr 172.21.15.79:0/1267909291 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:32.098 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48401930 0 0) 0x7f5a30071410 con 0x7f5a3006fe90 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient(hunting): _negotiate 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c003520 con 0x7f5a3006fe90 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965764388 0 0) 0x7f5a1c003520 con 0x7f5a3006fe90 2024-03-29T10:58:32.099 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 >> v1:172.21.15.79:6790/0 conn(0x7f5a30143c50 legacy=0x7f5a3014b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 >> v1:172.21.15.79:6789/0 conn(0x7f5a300708a0 legacy=0x7f5a300af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: _finish_hunting 0 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 monclient: found mon.noname-c 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.82:6789/0 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300ac8b0 con 0x7f5a3006fe90 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: _finish_auth 0 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: _check_auth_tickets 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5a20001670 con 0x7f5a3006fe90 2024-03-29T10:58:32.100 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a3504d640 10 monclient: dump: 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a382b3640 5 monclient: authenticate success, global_id 4543 2024-03-29T10:58:32.101 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.118+0000 7f5a382b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 <== mon.1 v1:172.21.15.82:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a20001840 con 0x7f5a3006fe90 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a3504d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a3504d640 1 -- 172.21.15.79:0/1267909291 <== mon.1 v1:172.21.15.82:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5a20001c50 con 0x7f5a3006fe90 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a3504d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a3504d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a3504d640 10 monclient: dump: 2024-03-29T10:58:32.102 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a382b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:32.103 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.122+0000 7f5a382b3640 4 set_mon_vals no callback set 2024-03-29T10:58:32.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.123+0000 7f5a382b3640 10 monclient: shutdown 2024-03-29T10:58:32.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.123+0000 7f5a382b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:32.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.123+0000 7f5a382b3640 1 -- 172.21.15.79:0/1267909291 >> v1:172.21.15.82:6789/0 conn(0x7f5a3006fe90 legacy=0x7f5a30143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:32.104 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.123+0000 7f5a382b3640 1 -- 172.21.15.79:0/1267909291 shutdown_connections 2024-03-29T10:58:32.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.123+0000 7f5a382b3640 1 -- 172.21.15.79:0/1267909291 >> 172.21.15.79:0/1267909291 conn(0x7f5a300675b0 msgr2=0x7f5a300b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:32.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.123+0000 7f5a382b3640 1 -- 172.21.15.79:0/1267909291 shutdown_connections 2024-03-29T10:58:32.105 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.124+0000 7f5a382b3640 1 -- 172.21.15.79:0/1267909291 wait complete. 2024-03-29T10:58:32.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.124+0000 7f5a382b3640 10 monclient: build_initial_monmap 2024-03-29T10:58:32.109 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.124+0000 7f5a382b3640 10 monclient: monmap: 2024-03-29T10:58:32.109 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:32.109 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:32.109 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.125+0000 7f5a382b3640 1 librados: starting msgr at 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.125+0000 7f5a382b3640 1 librados: starting objecter 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.125+0000 7f5a382b3640 1 Processor -- start 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.126+0000 7f5a382b3640 1 -- start start 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.126+0000 7f5a382b3640 1 librados: setting wanted keys 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.126+0000 7f5a382b3640 1 librados: calling monclient init 2024-03-29T10:58:32.110 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.126+0000 7f5a382b3640 10 monclient: init 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5a3006fe90 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient(hunting): picked mon.noname-c con 0x7f5a300708a0 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5a30143c50 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3013a0c0 con 0x7f5a30143c50 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3013d630 con 0x7f5a3006fe90 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3013a330 con 0x7f5a300708a0 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:32.111 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.127+0000 7f5a382b3640 10 monclient(hunting): authenticate will time out at 1631.419328s 2024-03-29T10:58:32.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a36850640 1 --1- >> v1:172.21.15.79:6789/0 conn(0x7f5a30143c50 0x7f5a300aa8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6789/0 says I am v1:172.21.15.79:52047/0 (socket says 172.21.15.79:52047) 2024-03-29T10:58:32.112 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a36850640 1 -- 172.21.15.79:0/126033082 learned_addr learned my addr 172.21.15.79:0/126033082 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:32.115 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 10 client.?.objecter ms_handle_connect 0x7f5a30143c50 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887739801 0 0) 0x7f5a3013a0c0 con 0x7f5a30143c50 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 10 monclient(hunting): my global_id is 4608 2024-03-29T10:58:32.116 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.130+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a14004eb0 con 0x7f5a30143c50 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1008227368 0 0) 0x7f5a14004eb0 con 0x7f5a30143c50 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a3013a0c0 con 0x7f5a30143c50 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5a28001c60 con 0x7f5a30143c50 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.120 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.131+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2959090391 0 0) 0x7f5a3013a0c0 con 0x7f5a30143c50 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.132+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 >> v1:172.21.15.79:6790/0 conn(0x7f5a300708a0 legacy=0x7f5a3013b600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 >> v1:172.21.15.82:6789/0 conn(0x7f5a3006fe90 legacy=0x7f5a3013aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 1 monclient: found mon.a 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3013a610 con 0x7f5a30143c50 2024-03-29T10:58:32.121 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 5 monclient: authenticate success, global_id 4608 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 10 monclient: _renew_subs 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a300ac5b0 con 0x7f5a30143c50 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 10 monclient: _renew_subs 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:32.122 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a30136760 con 0x7f5a30143c50 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a382b3640 1 librados: init done 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a28003970 con 0x7f5a30143c50 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f5a28003d80 con 0x7f5a30143c50 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.136+0000 7f5a26ffd640 10 monclient: dump: 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:32.123 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:32.124 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:32.124 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:32.124 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:32.125 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.138+0000 7f5a382b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.138+0000 7f5a382b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.138+0000 7f5a382b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.138+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30071e40 con 0x7f5a30143c50 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.139+0000 7f5a27fff640 4 set_mon_vals no callback set 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.140+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f5a28003ee0 con 0x7f5a30143c50 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.141+0000 7f5a3604f640 10 monclient: get_auth_request con 0x7f5a1407ec10 auth_method 0 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.142+0000 7f5a26ffd640 10 client.4608.objecter ms_handle_connect 0x7f5a1407ec10 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.142+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7f5a28084e10 con 0x7f5a30143c50 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.142+0000 7f5a26ffd640 10 client.4608.objecter ms_dispatch 0x7f5a30000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.142+0000 7f5a26ffd640 3 client.4608.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.142+0000 7f5a26ffd640 3 client.4608.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:32.126 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.142+0000 7f5a26ffd640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-29T10:58:32.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.149+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a280506f0 con 0x7f5a30143c50 2024-03-29T10:58:32.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.149+0000 7f5a26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:32.130 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.149+0000 7f5a26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:32.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.482+0000 7f5a382b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:32.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.482+0000 7f5a382b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:32.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.482+0000 7f5a382b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.79:6789/0 2024-03-29T10:58:32.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.482+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 --> v1:172.21.15.79:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a3013ab30 con 0x7f5a30143c50 2024-03-29T10:58:32.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.483+0000 7f5a26ffd640 1 -- 172.21.15.79:0/126033082 <== mon.0 v1:172.21.15.79:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7f5a3013ab30 con 0x7f5a30143c50 2024-03-29T10:58:32.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.483+0000 7f5a26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:32.463 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.483+0000 7f5a26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:32.466 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:32.466 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:32.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 10 librados: watch_flush enter 2024-03-29T10:58:32.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 10 librados: watch_flush exit 2024-03-29T10:58:32.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 10 client.4608.objecter successfully canceled tick 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 >> v1:172.21.15.79:6816/4290694627 conn(0x7f5a1407ec10 legacy=0x7f5a140810d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 10 monclient: shutdown 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.489+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 >> v1:172.21.15.79:6789/0 conn(0x7f5a30143c50 legacy=0x7f5a300aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.491+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 shutdown_connections 2024-03-29T10:58:32.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.491+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 >> 172.21.15.79:0/126033082 conn(0x7f5a300675b0 msgr2=0x7f5a300b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:32.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.491+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 shutdown_connections 2024-03-29T10:58:32.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.492+0000 7f5a382b3640 1 -- 172.21.15.79:0/126033082 wait complete. 2024-03-29T10:58:32.478 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:32.497+0000 7f5a382b3640 1 librados: shutdown 2024-03-29T10:58:32.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:39.308 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:39.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.479+0000 7f9b48a8d640 10 monclient: get_monmap_and_config 2024-03-29T10:58:39.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.479+0000 7f9b48a8d640 10 monclient: build_initial_monmap 2024-03-29T10:58:39.460 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.479+0000 7f9b48a8d640 10 monclient: monmap: 2024-03-29T10:58:39.460 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:39.460 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:39.480281+0000 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:39.480281+0000 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:39.461 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:39.462 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.482+0000 7f9b48a8d640 1 Processor -- start 2024-03-29T10:58:39.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.483+0000 7f9b48a8d640 1 -- start start 2024-03-29T10:58:39.465 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.483+0000 7f9b48a8d640 10 monclient: init 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b440a9ac0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b4413ee90 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b440af900 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440afd10 con 0x7f9b440a9ac0 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440af6f0 con 0x7f9b440af900 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440aac80 con 0x7f9b4413ee90 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.484+0000 7f9b48a8d640 10 monclient(hunting): authenticate will time out at 1638.777050s 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.485+0000 7f9b41d9b640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f9b4413ee90 0x7f9b4413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:52778/0 (socket says 172.21.15.79:52778) 2024-03-29T10:58:39.466 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.485+0000 7f9b41d9b640 1 -- 172.21.15.79:0/2465037662 learned_addr learned my addr 172.21.15.79:0/2465037662 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506694181 0 0) 0x7f9b440aac80 con 0x7f9b4413ee90 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 10 monclient(hunting): _negotiate 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 10 monclient(hunting): my global_id is 4694 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b2c003520 con 0x7f9b4413ee90 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003750577 0 0) 0x7f9b2c003520 con 0x7f9b4413ee90 2024-03-29T10:58:39.467 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.486+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 >> v1:172.21.15.82:6789/0 conn(0x7f9b440af900 legacy=0x7f9b44149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 >> v1:172.21.15.79:6789/0 conn(0x7f9b440a9ac0 legacy=0x7f9b440adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: _finish_hunting 0 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 1 monclient: found mon.noname-b 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440aaf60 con 0x7f9b4413ee90 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: _finish_auth 0 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: _check_auth_tickets 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9b34001690 con 0x7f9b4413ee90 2024-03-29T10:58:39.470 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.489+0000 7f9b4159a640 10 monclient: dump: 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:39.471 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:39.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b340018f0 con 0x7f9b4413ee90 2024-03-29T10:58:39.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b4159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:39.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b4159a640 1 -- 172.21.15.79:0/2465037662 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9b34001d40 con 0x7f9b4413ee90 2024-03-29T10:58:39.472 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b4159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b4159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b4159a640 10 monclient: dump: 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b48a8d640 5 monclient: authenticate success, global_id 4694 2024-03-29T10:58:39.473 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b48a8d640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.490+0000 7f9b48a8d640 4 set_mon_vals no callback set 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.491+0000 7f9b48a8d640 10 monclient: shutdown 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.491+0000 7f9b48a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.491+0000 7f9b48a8d640 1 -- 172.21.15.79:0/2465037662 >> v1:172.21.15.79:6790/0 conn(0x7f9b4413ee90 legacy=0x7f9b4413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.492+0000 7f9b48a8d640 1 -- 172.21.15.79:0/2465037662 shutdown_connections 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.492+0000 7f9b48a8d640 1 -- 172.21.15.79:0/2465037662 >> 172.21.15.79:0/2465037662 conn(0x7f9b440b35e0 msgr2=0x7f9b440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:39.474 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.492+0000 7f9b48a8d640 1 -- 172.21.15.79:0/2465037662 shutdown_connections 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.492+0000 7f9b48a8d640 1 -- 172.21.15.79:0/2465037662 wait complete. 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.493+0000 7f9b48a8d640 10 monclient: build_initial_monmap 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.493+0000 7f9b48a8d640 10 monclient: monmap: 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:39.475 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.494+0000 7f9b48a8d640 1 librados: starting msgr at 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.494+0000 7f9b48a8d640 1 librados: starting objecter 2024-03-29T10:58:39.476 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.494+0000 7f9b48a8d640 1 Processor -- start 2024-03-29T10:58:39.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.496+0000 7f9b48a8d640 1 -- start start 2024-03-29T10:58:39.479 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.496+0000 7f9b48a8d640 1 librados: setting wanted keys 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.496+0000 7f9b48a8d640 1 librados: calling monclient init 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.496+0000 7f9b48a8d640 10 monclient: init 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b440a9ac0 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b440af900 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:39.480 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b4413ee90 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b441e40f0 con 0x7f9b4413ee90 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440716b0 con 0x7f9b440a9ac0 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b44071e30 con 0x7f9b440af900 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.497+0000 7f9b48a8d640 10 monclient(hunting): authenticate will time out at 1638.789284s 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.499+0000 7f9b41d9b640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f9b440af900 0x7f9b441e3350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:55094/0 (socket says 172.21.15.79:55094) 2024-03-29T10:58:39.481 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.499+0000 7f9b41d9b640 1 -- 172.21.15.79:0/3187556330 learned_addr learned my addr 172.21.15.79:0/3187556330 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:39.482 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 client.?.objecter ms_handle_connect 0x7f9b440af900 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420866493 0 0) 0x7f9b44071e30 con 0x7f9b440af900 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 monclient(hunting): my global_id is 4697 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b24004eb0 con 0x7f9b440af900 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 client.?.objecter ms_handle_connect 0x7f9b4413ee90 2024-03-29T10:58:39.483 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.500+0000 7f9b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:39.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 client.?.objecter ms_handle_connect 0x7f9b440a9ac0 2024-03-29T10:58:39.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:39.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832839167 0 0) 0x7f9b441e40f0 con 0x7f9b4413ee90 2024-03-29T10:58:39.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:39.484 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 monclient(hunting): my global_id is 4611 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b44071e30 con 0x7f9b4413ee90 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219065679 0 0) 0x7f9b24004eb0 con 0x7f9b440af900 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b441e40f0 con 0x7f9b440af900 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9b34001650 con 0x7f9b440af900 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.501+0000 7f9b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:39.485 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366350302 0 0) 0x7f9b440716b0 con 0x7f9b440a9ac0 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:39.486 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b24004eb0 con 0x7f9b440a9ac0 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2363050668 0 0) 0x7f9b441e40f0 con 0x7f9b440af900 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 >> v1:172.21.15.82:6789/0 conn(0x7f9b440a9ac0 legacy=0x7f9b44070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 >> v1:172.21.15.79:6789/0 conn(0x7f9b4413ee90 legacy=0x7f9b441ef930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 monclient: found mon.c 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:39.487 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b44072110 con 0x7f9b440af900 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.502+0000 7f9b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b34004a90 con 0x7f9b440af900 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f9b34004da0 con 0x7f9b440af900 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b32ffd640 10 monclient: dump: 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:39.488 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:39.489 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:39.489 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:39.489 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:39.489 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:39.489 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.503+0000 7f9b33fff640 4 set_mon_vals no callback set 2024-03-29T10:58:39.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 5 monclient: authenticate success, global_id 4697 2024-03-29T10:58:39.490 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 10 monclient: _renew_subs 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b440723f0 con 0x7f9b440af900 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 10 monclient: _renew_subs 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b44070e20 con 0x7f9b440af900 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.504+0000 7f9b48a8d640 1 librados: init done 2024-03-29T10:58:39.491 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.506+0000 7f9b48a8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.506+0000 7f9b48a8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.506+0000 7f9b48a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.506+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b440710c0 con 0x7f9b440af900 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.507+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f9b3401ad70 con 0x7f9b440af900 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.508+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7f9b34084b60 con 0x7f9b440af900 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.508+0000 7f9b32ffd640 10 client.4697.objecter ms_dispatch 0x7f9b44000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.508+0000 7f9b32ffd640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.508+0000 7f9b32ffd640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.508+0000 7f9b32ffd640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T10:58:39.492 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.508+0000 7f9b4259c640 10 monclient: get_auth_request con 0x7f9b2406dc60 auth_method 0 2024-03-29T10:58:39.493 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.509+0000 7f9b32ffd640 10 client.4697.objecter ms_handle_connect 0x7f9b2406dc60 2024-03-29T10:58:39.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.516+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b34050440 con 0x7f9b440af900 2024-03-29T10:58:39.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.516+0000 7f9b32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:39.496 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.516+0000 7f9b32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:39.826 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.845+0000 7f9b48a8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:39.826 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.845+0000 7f9b48a8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:39.826 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.845+0000 7f9b48a8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:39.826 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.845+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b4413bd00 con 0x7f9b440af900 2024-03-29T10:58:39.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.846+0000 7f9b32ffd640 1 -- 172.21.15.79:0/3187556330 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7f9b4413bd00 con 0x7f9b440af900 2024-03-29T10:58:39.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.846+0000 7f9b32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:39.827 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.846+0000 7f9b32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:39.827 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:39.827 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:39.831 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.850+0000 7f9b48a8d640 10 librados: watch_flush enter 2024-03-29T10:58:39.831 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.850+0000 7f9b48a8d640 10 librados: watch_flush exit 2024-03-29T10:58:39.831 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.850+0000 7f9b48a8d640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T10:58:39.831 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.850+0000 7f9b48a8d640 10 client.4697.objecter successfully canceled tick 2024-03-29T10:58:39.832 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.851+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 >> v1:172.21.15.79:6816/4290694627 conn(0x7f9b2406dc60 legacy=0x7f9b24070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:39.832 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.851+0000 7f9b48a8d640 10 monclient: shutdown 2024-03-29T10:58:39.832 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.851+0000 7f9b48a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:39.832 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.851+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 >> v1:172.21.15.79:6790/0 conn(0x7f9b440af900 legacy=0x7f9b441e3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:39.832 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.851+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 shutdown_connections 2024-03-29T10:58:39.832 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.851+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 >> 172.21.15.79:0/3187556330 conn(0x7f9b440b35e0 msgr2=0x7f9b440adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:39.833 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.852+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 shutdown_connections 2024-03-29T10:58:39.833 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.852+0000 7f9b48a8d640 1 -- 172.21.15.79:0/3187556330 wait complete. 2024-03-29T10:58:39.833 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:39.852+0000 7f9b48a8d640 1 librados: shutdown 2024-03-29T10:58:39.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:46.662 DEBUG:teuthology.orchestra.run.smithi079:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.819+0000 7f59ef7ba640 10 monclient: get_monmap_and_config 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.819+0000 7f59ef7ba640 10 monclient: build_initial_monmap 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.819+0000 7f59ef7ba640 10 monclient: monmap: 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:58:46.820703+0000 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:58:46.820703+0000 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-c 2024-03-29T10:58:46.800 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-b 2024-03-29T10:58:46.801 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:46.801 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.820+0000 7f59ef7ba640 1 Processor -- start 2024-03-29T10:58:46.802 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.821+0000 7f59ef7ba640 1 -- start start 2024-03-29T10:58:46.802 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.821+0000 7f59ef7ba640 10 monclient: init 2024-03-29T10:58:46.802 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.821+0000 7f59ef7ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:46.802 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.821+0000 7f59ef7ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:46.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 10 monclient(hunting): picked mon.noname-c con 0x7f59e81457f0 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:46.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 10 monclient(hunting): picked mon.noname-b con 0x7f59e813f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:46.803 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 10 monclient(hunting): picked mon.noname-a con 0x7f59e80a9d70 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e80aaef0 con 0x7f59e80a9d70 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8145bd0 con 0x7f59e81457f0 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8145eb0 con 0x7f59e813f650 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ef7ba640 10 monclient(hunting): authenticate will time out at 1646.114791s 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ecd55640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f59e813f650 0x7f59e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:38078/0 (socket says 172.21.15.79:38078) 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.822+0000 7f59ecd55640 1 -- 172.21.15.79:0/1268231683 learned_addr learned my addr 172.21.15.79:0/1268231683 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326151901 0 0) 0x7f59e8145eb0 con 0x7f59e813f650 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): my global_id is 4700 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d4003520 con 0x7f59e813f650 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 <== mon.0 v1:172.21.15.79:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937168295 0 0) 0x7f59e80aaef0 con 0x7f59e80a9d70 2024-03-29T10:58:46.804 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:46.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:46.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:46.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 10 monclient(hunting): my global_id is 4614 2024-03-29T10:58:46.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 --> v1:172.21.15.79:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e8145eb0 con 0x7f59e80a9d70 2024-03-29T10:58:46.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 784965614 0 0) 0x7f59d4003520 con 0x7f59e813f650 2024-03-29T10:58:46.805 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.823+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 >> v1:172.21.15.82:6789/0 conn(0x7f59e81457f0 legacy=0x7f59e813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:46.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 >> v1:172.21.15.79:6789/0 conn(0x7f59e80a9d70 legacy=0x7f59e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:46.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:46.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 1 monclient: found mon.noname-b 2024-03-29T10:58:46.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:46.806 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.79:6790/0 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8146340 con 0x7f59e813f650 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f59d8001690 con 0x7f59e813f650 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: dump: 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:46.807 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59ef7ba640 5 monclient: authenticate success, global_id 4700 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59ef7ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 <== mon.2 v1:172.21.15.79:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d80017f0 con 0x7f59e813f650 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 1 -- 172.21.15.79:0/1268231683 <== mon.2 v1:172.21.15.79:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f59d8001c40 con 0x7f59e813f650 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59dffff640 10 monclient: dump: 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:46.808 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59ef7ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.824+0000 7f59ef7ba640 4 set_mon_vals no callback set 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.825+0000 7f59ef7ba640 10 monclient: shutdown 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.825+0000 7f59ef7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.825+0000 7f59ef7ba640 1 -- 172.21.15.79:0/1268231683 >> v1:172.21.15.79:6790/0 conn(0x7f59e813f650 legacy=0x7f59e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.825+0000 7f59ef7ba640 1 -- 172.21.15.79:0/1268231683 shutdown_connections 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.825+0000 7f59ef7ba640 1 -- 172.21.15.79:0/1268231683 >> 172.21.15.79:0/1268231683 conn(0x7f59e8066a00 msgr2=0x7f59e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:46.809 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.826+0000 7f59ef7ba640 1 -- 172.21.15.79:0/1268231683 shutdown_connections 2024-03-29T10:58:46.810 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.826+0000 7f59ef7ba640 1 -- 172.21.15.79:0/1268231683 wait complete. 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 10 monclient: build_initial_monmap 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 10 monclient: monmap: 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:epoch 0 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:last_changed 0.000000 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:created 0.000000 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.noname-a 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.noname-b 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.noname-c 2024-03-29T10:58:46.811 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 1 librados: starting msgr at 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 1 librados: starting objecter 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 1 Processor -- start 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 1 -- start start 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 1 librados: setting wanted keys 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 1 librados: calling monclient init 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.827+0000 7f59ef7ba640 10 monclient: init 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.828+0000 7f59ef7ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:46.812 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.828+0000 7f59ef7ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:46.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 10 monclient(hunting): picked mon.noname-b con 0x7f59e80a9d70 addr v1:172.21.15.82:6789/0 2024-03-29T10:58:46.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 10 monclient(hunting): picked mon.noname-c con 0x7f59e813f650 addr v1:172.21.15.79:6790/0 2024-03-29T10:58:46.814 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 10 monclient(hunting): picked mon.noname-a con 0x7f59e81457f0 addr v1:172.21.15.79:6789/0 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 1 -- --> v1:172.21.15.79:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8144620 con 0x7f59e81457f0 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8141370 con 0x7f59e80a9d70 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 1 -- --> v1:172.21.15.79:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e8141650 con 0x7f59e813f650 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ef7ba640 10 monclient(hunting): authenticate will time out at 1646.123438s 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ecd55640 1 --1- >> v1:172.21.15.79:6790/0 conn(0x7f59e813f650 0x7f59e81dedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.79:6790/0 says I am v1:172.21.15.79:39234/0 (socket says 172.21.15.79:39234) 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ecd55640 1 -- 172.21.15.79:0/2629640660 learned_addr learned my addr 172.21.15.79:0/2629640660 (peer_addr_for_me v1:172.21.15.79:0/0) 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 client.?.objecter ms_handle_connect 0x7f59e813f650 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964323783 0 0) 0x7f59e8141650 con 0x7f59e813f650 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:46.815 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): my global_id is 4703 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc00d6c0 con 0x7f59e813f650 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4168690114 0 0) 0x7f59cc00d6c0 con 0x7f59e813f650 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e8141650 con 0x7f59e813f650 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f59d8001bc0 con 0x7f59e813f650 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): dump: 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:46.816 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1031012074 0 0) 0x7f59e8141650 con 0x7f59e813f650 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.831+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 >> v1:172.21.15.82:6789/0 conn(0x7f59e80a9d70 legacy=0x7f59e813fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 >> v1:172.21.15.79:6789/0 conn(0x7f59e81457f0 legacy=0x7f59e81e2500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 1 monclient: found mon.c 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e813c650 con 0x7f59e813f650 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:46.817 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ef7ba640 5 monclient: authenticate success, global_id 4703 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ef7ba640 10 monclient: _renew_subs 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ef7ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e81404b0 con 0x7f59e813f650 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d8004bf0 con 0x7f59e813f650 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1946848729 0 0) 0x7f59d8004f00 con 0x7f59e813f650 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59ddffb640 10 monclient: dump: 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:epoch 1 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:fsid 13c29107-6abd-40ad-99ae-e45192d5c573 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:last_changed 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:created 2024-03-29T10:47:10.744548+0000 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:46.818 INFO:teuthology.misc.health.smithi079.stderr:election_strategy: 1 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:0: v1:172.21.15.79:6789/0 mon.a 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:1: v1:172.21.15.82:6789/0 mon.b 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2: v1:172.21.15.79:6790/0 mon.c 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr: 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.832+0000 7f59df7fe640 4 set_mon_vals no callback set 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.833+0000 7f59ef7ba640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.833+0000 7f59ef7ba640 10 monclient: _renew_subs 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.833+0000 7f59ef7ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.833+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e81452c0 con 0x7f59e813f650 2024-03-29T10:58:46.819 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.833+0000 7f59ef7ba640 1 librados: init done 2024-03-29T10:58:46.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.834+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2092610235 0 0) 0x7f59d801aed0 con 0x7f59e813f650 2024-03-29T10:58:46.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ef7ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:46.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ef7ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:46.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ef7ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:46.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e81447d0 con 0x7f59e813f650 2024-03-29T10:58:46.820 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 921230513 0 0) 0x7f59d8085d90 con 0x7f59e813f650 2024-03-29T10:58:46.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ddffb640 10 client.4703.objecter ms_dispatch 0x7f59e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:58:46.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ddffb640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:58:46.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ddffb640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:58:46.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ddffb640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-29T10:58:46.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ed556640 10 monclient: get_auth_request con 0x7f59cc0763c0 auth_method 0 2024-03-29T10:58:46.821 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.835+0000 7f59ddffb640 10 client.4703.objecter ms_handle_connect 0x7f59cc0763c0 2024-03-29T10:58:46.822 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.842+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59e81447d0 con 0x7f59e813f650 2024-03-29T10:58:46.822 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.842+0000 7f59ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:46.822 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:46.842+0000 7f59ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:47.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.155+0000 7f59ef7ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:47.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.156+0000 7f59ef7ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:47.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.156+0000 7f59ef7ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.79:6790/0 2024-03-29T10:58:47.136 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.156+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 --> v1:172.21.15.79:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59e81447d0 con 0x7f59e813f650 2024-03-29T10:58:47.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.156+0000 7f59ddffb640 1 -- 172.21.15.79:0/2629640660 <== mon.2 v1:172.21.15.79:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3487261848) 0x7f59e81447d0 con 0x7f59e813f650 2024-03-29T10:58:47.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.156+0000 7f59ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:47.137 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.156+0000 7f59ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:47.137 INFO:teuthology.misc.health.smithi079.stdout: 2024-03-29T10:58:47.137 INFO:teuthology.misc.health.smithi079.stdout:{"epoch":11,"fsid":"13c29107-6abd-40ad-99ae-e45192d5c573","created":"2024-03-29T10:47:48.928709+0000","modified":"2024-03-29T10:57:53.851689+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:57:53.851689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e014f81f-8d58-44ef-8d6e-2c0895a7c3bd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"3dcd6634-e6ea-4862-aef6-f70ff756ca74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f3cdacad-8901-4794-ad1c-1368e5a42e0a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cfba379b-8173-47fb-b761-87462a52e762","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"729faf03-f69a-40f8-9ae9-1b98771fa643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"18c4cbac-2e00-43dd-ba9d-15754f0a644a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00c2423f-5d63-4648-a44b-2fadd7698e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae0947ca-ffcb-4179-8a37-b246ab449ff4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:47.140 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.160+0000 7f59ef7ba640 10 librados: watch_flush enter 2024-03-29T10:58:47.140 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 10 librados: watch_flush exit 2024-03-29T10:58:47.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-29T10:58:47.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 10 client.4703.objecter successfully canceled tick 2024-03-29T10:58:47.141 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 >> v1:172.21.15.79:6816/4290694627 conn(0x7f59cc0763c0 legacy=0x7f59cc078880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 10 monclient: shutdown 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 >> v1:172.21.15.79:6790/0 conn(0x7f59e813f650 legacy=0x7f59e81dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 shutdown_connections 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 >> 172.21.15.79:0/2629640660 conn(0x7f59e8066a00 msgr2=0x7f59e80b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 shutdown_connections 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 -- 172.21.15.79:0/2629640660 wait complete. 2024-03-29T10:58:47.142 INFO:teuthology.misc.health.smithi079.stderr:2024-03-29T10:58:47.161+0000 7f59ef7ba640 1 librados: shutdown 2024-03-29T10:58:47.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:47.157 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:58:47.158 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:58:47.159 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:58:47.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.196 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:58:47.196 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:58:47.196 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.269 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:58:47.270 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:58:47.270 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.337 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:58:47.337 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:58:47.338 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.411 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:58:47.412 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:58:47.412 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.448 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:58:47.448 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:58:47.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.521 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:58:47.521 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:58:47.521 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.592 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:58:47.592 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:58:47.592 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.658 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:58:47.658 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:58:47.658 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:58:47.658 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.683 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:58:47.683 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:58:47.683 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.723 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:58:47.723 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:58:47.724 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:58:47.724 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.736 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:58:47.736 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:58:47.736 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.790 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:58:47.790 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:58:47.790 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:58:47.801 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:58:47.801 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:58:47.801 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:58:47.966 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:58:47.966 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:58:54.350 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:58:54.350 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:58:54.459 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:58:54.459 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:58:54.527 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T10:58:54.527 DEBUG:teuthology.orchestra.run.smithi079:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:58:54.587 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:58:54.587 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:58:55.292 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:58:55.292 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:58:55.421 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:58:55.421 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:58:55.480 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T10:58:55.481 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:58:55.538 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:58:55.540 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/data/mon.a.tgz 2024-03-29T10:58:55.541 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-03-29T10:58:55.558 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.6Gq2bOaqyL 2024-03-29T10:58:55.558 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6Gq2bOaqyL 2024-03-29T10:58:55.660 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.6Gq2bOaqyL 2024-03-29T10:58:55.748 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.6Gq2bOaqyL is 126KB 2024-03-29T10:58:55.800 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.6Gq2bOaqyL 2024-03-29T10:58:55.815 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/data/mon.c.tgz 2024-03-29T10:58:55.815 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-03-29T10:58:55.870 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.B7KzyHNYDf 2024-03-29T10:58:55.870 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.B7KzyHNYDf 2024-03-29T10:58:55.960 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.B7KzyHNYDf 2024-03-29T10:58:56.119 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.B7KzyHNYDf is 126KB 2024-03-29T10:58:56.163 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.B7KzyHNYDf 2024-03-29T10:58:56.180 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/data/mon.b.tgz 2024-03-29T10:58:56.181 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-03-29T10:58:56.201 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.T558y7tQvo 2024-03-29T10:58:56.201 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.T558y7tQvo 2024-03-29T10:58:56.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.T558y7tQvo 2024-03-29T10:58:56.438 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.T558y7tQvo is 126KB 2024-03-29T10:58:56.549 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.T558y7tQvo 2024-03-29T10:58:56.566 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:58:56.566 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:58:56.569 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:58:56.651 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:58:56.689 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:58:56.689 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:58:56.819 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/remote/smithi079/crash 2024-03-29T10:58:56.821 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:58:56.849 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/remote/smithi082/crash 2024-03-29T10:58:56.849 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:58:56.877 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:58:56.877 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:58:56.893 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:58:56.918 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2024-03-29T10:58:56.918 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:58:56.963 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:58:56.963 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34741.log 2024-03-29T10:58:56.963 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:58:56.963 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34621.log 2024-03-29T10:58:56.963 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34461.log 2024-03-29T10:58:56.963 INFO:teuthology.orchestra.run.smithi082.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34741.log.gz 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34621.log: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34621.log.gz 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34701.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34461.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34701.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34461.log.gz 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34701.log.gz 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34661.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mgr.x.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34581.log 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-03-29T10:58:56.964 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34501.log 2024-03-29T10:58:56.965 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34581.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:58:56.965 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34501.log: 87.8% 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:58:56.965 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34581.log.gz 2024-03-29T10:58:56.965 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34541.log 2024-03-29T10:58:56.965 INFO:teuthology.orchestra.run.smithi082.stderr: 87.7%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph-client.admin.34501.log.gz 2024-03-29T10:58:56.965 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.34541.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34541.log.gz 2024-03-29T10:58:56.966 INFO:teuthology.orchestra.run.smithi082.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:58:56.973 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38576.log 2024-03-29T10:58:56.973 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41114.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40295.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39905.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38576.log.gz 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41231.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40295.log.gz 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39593.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39905.log.gz 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40529.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38615.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34878.log 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41231.log.gz 2024-03-29T10:58:56.974 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38615.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40529.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38693.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38615.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40685.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34878.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38693.log: gzip 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38693.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39710.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-03-29T10:58:56.975 INFO:teuthology.orchestra.run.smithi079.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41543.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39710.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40256.log 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41699.log 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40412.log 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40256.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log/var/log/ceph/ceph-client.admin.41699.log: 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40724.log 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41699.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40412.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39476.log 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-03-29T10:58:56.976 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38418.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39476.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39944.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38418.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39476.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40139.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38967.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38418.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39944.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38967.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40139.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40061.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38379.log 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2024-03-29T10:58:56.977 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41192.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40061.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38849.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38379.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39749.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38379.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.audit.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38849.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39045.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38537.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39045.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39749.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39045.log.gz 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40958.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38537.log: /var/log/ceph/ceph-client.admin.38771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2024-03-29T10:58:56.978 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38810.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38537.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34982.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39983.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38810.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40451.log: /var/log/ceph/ceph-client.admin.39983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41075.log 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39983.log.gz 2024-03-29T10:58:56.979 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41075.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39632.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40373.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39357.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41075.log.gz 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40763.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40373.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39632.log.gz 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38457.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40373.log.gz 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40763.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40022.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38457.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40763.log.gz 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40217.log 2024-03-29T10:58:56.980 INFO:teuthology.orchestra.run.smithi079.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38457.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40997.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38732.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40217.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40022.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40217.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41582.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40997.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39084.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38732.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38927.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41582.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41270.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41582.log.gz 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38927.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:58:56.981 INFO:teuthology.orchestra.run.smithi079.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39006.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38927.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39515.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41270.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41387.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39515.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39006.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41504.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41387.log: gzip -5 --verbose -- 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39515.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr: /var/log/ceph/ceph-client.admin.38888.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41504.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40178.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41504.log.gz 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38262.log 2024-03-29T10:58:56.982 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40178.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39671.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41348.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40178.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39866.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38262.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38496.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39671.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log/var/log/ceph/ceph-client.admin.39866.log: 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41348.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38496.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28776.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39866.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.0.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38496.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40880.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.tmp-client.admin.28776.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28776.log.gz 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41309.log 2024-03-29T10:58:56.983 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40802.log: gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40880.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39788.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41309.log: /var/log/ceph/ceph-client.admin.39788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39788.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41309.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40568.log: /var/log/ceph/ceph-client.admin.39162.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38340.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41153.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39437.log 2024-03-29T10:58:56.984 INFO:teuthology.orchestra.run.smithi079.stderr: 85.7%/var/log/ceph/ceph-client.admin.41153.log: -- replaced with /var/log/ceph/ceph-client.admin.38340.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39437.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41153.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39437.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34587.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38301.log: -- /var/log/ceph/ceph-client.admin.39240.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.34587.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.39240.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34587.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr: -- /var/log/ceph/ceph-client.admin.41426.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40100.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41426.log: /var/log/ceph/ceph-client.admin.39201.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41426.log.gz 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-03-29T10:58:56.985 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40100.log.gz 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41660.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40607.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41465.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41660.log: -- /var/log/ceph/ceph-client.admin.38654.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.41465.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41660.log.gz 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40841.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.38654.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41465.log.gz 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38654.log.gz 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40841.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:58:56.986 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-client.admin.40919.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40841.log.gz 2024-03-29T10:58:56.987 INFO:teuthology.orchestra.run.smithi079.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz 2024-03-29T10:58:57.008 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/ceph-mon.c.log: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:59:08.570 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:59:09.165 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:59:09.438 INFO:teuthology.orchestra.run.smithi079.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:59:12.613 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:59:13.171 INFO:teuthology.orchestra.run.smithi079.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:59:13.252 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:59:13.347 INFO:teuthology.orchestra.run.smithi079.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:59:14.162 INFO:teuthology.orchestra.run.smithi079.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:59:44.327 INFO:teuthology.orchestra.run.smithi079.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:59:56.426 INFO:teuthology.orchestra.run.smithi079.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:59:56.428 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-03-29T10:59:56.428 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m59.522s 2024-03-29T10:59:56.428 INFO:teuthology.orchestra.run.smithi079.stderr:user 2m43.163s 2024-03-29T10:59:56.428 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m5.719s 2024-03-29T11:00:03.311 INFO:teuthology.orchestra.run.smithi082.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:00:03.312 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-29T11:00:03.312 INFO:teuthology.orchestra.run.smithi082.stderr:real 1m6.374s 2024-03-29T11:00:03.312 INFO:teuthology.orchestra.run.smithi082.stderr:user 1m57.646s 2024-03-29T11:00:03.313 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m4.311s 2024-03-29T11:00:03.313 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:00:03.313 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/remote/smithi079/log 2024-03-29T11:00:03.314 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:00:26.709 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/remote/smithi082/log 2024-03-29T11:00:26.710 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:00:44.447 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:00:44.652 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e7bb36f42d6549b9bf2115d42c57f2bb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:00:44.655 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:00:44.672 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:00:44.686 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:00:44.686 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:00:44.704 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:00:44.783 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:00:44.791 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:00:44.792 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:00:44.794 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:00:44.809 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:00:44.810 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:00:44.813 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:00:44.813 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-03-29T11:00:44.813 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +438us[ +439us] +/- 78ms 2024-03-29T11:00:44.814 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -177us[ -176us] +/- 58ms 2024-03-29T11:00:44.814 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -202us[ -202us] +/- 88ms 2024-03-29T11:00:44.814 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:00:44.815 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:00:44.815 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-29T11:00:44.816 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +438us[ +428us] +/- 78ms 2024-03-29T11:00:44.816 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -184us[ -194us] +/- 58ms 2024-03-29T11:00:44.816 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -196us[ -196us] +/- 88ms 2024-03-29T11:00:44.816 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:00:44.816 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:00:44.825 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:00:44.825 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:00:44.834 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:00:44.928 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:00:44.995 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:00:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:45.028 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:00:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:45.058 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:00:45.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:00:45.075 INFO:teuthology.task.internal:Duration was 1282.588371 seconds 2024-03-29T11:00:45.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:00:45.084 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:00:45.084 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:00:45.087 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:00:45.129 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:00:45.142 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:00:45.555 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:00:45.555 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-03-29T11:00:45.555 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:00:45.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-03-29T11:00:45.576 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:00:45.618 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:00:45.619 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:00:45.621 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:00:45.686 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:00:45.687 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:00:45.692 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:00:45.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:00:45.836 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:00:45.836 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:00:45.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:00:45.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:00:45.899 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:45.906 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:45.929 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-03-29T11:00:45.955 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-03-29T11:00:45.980 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:46.020 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:00:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:00:46.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:00:46.045 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:00:46.046 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/remote/smithi079 2024-03-29T11:00:46.047 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:00:46.097 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651/remote/smithi082 2024-03-29T11:00:46.097 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:00:46.132 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:00:46.132 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:00:46.135 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:00:46.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:00:46.192 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:00:46.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:00:46.202 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:00:46.203 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:00:46.205 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:00:46.217 INFO:teuthology.orchestra.run.smithi079.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:00 /home/ubuntu/cephtest 2024-03-29T11:00:46.235 INFO:teuthology.orchestra.run.smithi082.stdout: 265799 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:00 /home/ubuntu/cephtest 2024-03-29T11:00:46.237 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:00:46.247 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:00:46.265 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:00:46.290 DEBUG:teuthology.nuke:shortname: smithi079 2024-03-29T11:00:46.291 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:00:46.315 DEBUG:teuthology.nuke:shortname: smithi082 2024-03-29T11:00:46.315 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:00:46.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:28:35.707724', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPN+qhibP5MMVarEo4s+gy8Sm4/97TldBKn8eK5G+fp0IKtrXNwB1gabHb7cTMmUTPY2wNZf3cU4EgfTWGG5M34='} 2024-03-29T11:00:46.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622651', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:28:35.706312', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLn8du5iTM/9h2XN3fmByzGpl7IBQMOUJR3iOwvLiEQM1xEK7UgdfKPxV2SN95N9JoBj4dZl5JKZauekhccJMRE='} 2024-03-29T11:00:46.370 INFO:teuthology.orchestra.console.smithi079:Power off 2024-03-29T11:00:46.370 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:00:46.397 DEBUG:teuthology.orchestra.console.smithi079:power off output: Chassis Power Control: Down/Off 2024-03-29T11:00:46.397 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:46.412 INFO:teuthology.orchestra.console.smithi082:Power off 2024-03-29T11:00:46.413 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:00:46.421 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-29T11:00:46.435 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-03-29T11:00:46.436 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:46.459 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-29T11:00:50.422 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:50.460 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:50.533 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is on 2024-03-29T11:00:50.570 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-29T11:00:54.538 DEBUG:teuthology.orchestra.console.smithi079:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:54.571 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:54.648 DEBUG:teuthology.orchestra.console.smithi079:check power output: Chassis Power is off 2024-03-29T11:00:54.680 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-29T11:00:54.749 INFO:teuthology.orchestra.console.smithi079:Power off completed 2024-03-29T11:00:58.684 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:00:58.795 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-03-29T11:00:58.896 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-03-29T11:00:59.000 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1282.5883705615997 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e7bb36f42d6549b9bf2115d42c57f2bb status: fail success: false 2024-03-29T11:00:59.000 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:00:59.076 INFO:teuthology.run:FAIL